You can do it with a tree too, but nobody really does it that way because there's not much reason to, other than academic curiosity. I'm not trying to stifle your ambition though.

Well, I tried out creating a tree to get rid of duplicate vertices, and it is indeed much faster. However, funny enough, keeping the tree balanced actually greatly decreases the speed. I guess, as the tree gets large enough, trying to keep it balanced just gets so unmanageable as it you have to rebalance it many, many, many times on your way up. I got an infinite recursion with one of my models, I'm guessing it was impossible to rebalance it to the threshold I had it at. Increasing the threashold seemed to do the trick, but was still much slower. (and when I say much slower, I mean a minute or two versus a couple of seconds)

With my tree implemented, I could open the 13 MB obj file in 2 seconds. After that, it takes about 6 seconds, I believe because of all the displaced memory from the first load. (it takes like 400 MB!) For comparison, a first-time open for Blender takes 8 seconds. Able to beat Blender with an import with my first obj loader? Not bad, I'd say. However, to be fare, I didn't load the texture files, but I don't think 2 tiff files can account for a 6 second difference. This is compared to several minutes (maybe as much as 10) with a linked list. I'd say that binary tree is certainly the way to go for vertex optimizing.

Edit: BTW, it brought the total vertices of that model from around 401979 to 138909

There is really no point in trying to remove dupes from .obj files. Streaming arrays is probably the fastest way of getting things to the GPU, so any magic to reduce the memory footprint will decrease performance. I wouldn't call it optimizing when it wrecks performance.

The arrays aren't necessarily the same size, and there are different indices for each face, which means that vertex arrays are worthless.

Perhaps you're misreading my posts. I'm getting each face, and creating a separate vertex for it. (since the vertex, color (when you resolve the material files), normal, and texture coordinate arrays aren't parallel) I then insert them into the tree, which will automatically get rid of duplicates, and returns the index used to put it into the vertex array. When it's done, I delete the tree and put the info into the actual vertex, normal, color, texture coordinate etc. arrays. When that's done, I then use the standard indexed vertex arrays to view the model, and will also export those arrays (along with other info) to the disk.

Three not-so-important points, because I'm sure you've already considered them:

1) A 13 MB obj file is rather large! Good for testing though I suppose.
2) This operation should not be done on load, only on export from your graphics package plug-in or script, or pre-converted on disk with a little utility app. The game that uses it should simply load the distilled product and not dink around with optimizations on load.
3) I am very surprised that the obj file wasn't optimized to get rid of duplicate vertices to begin with. Not saying they didn't legitimately need to be fixed, but I have my suspicions...

AnotherJake Wrote:Three not-so-important points, because I'm sure you've already considered them:

1) A 13 MB obj file is rather large! Good for testing though I suppose.

Yes, I know it's very large, and I was using it for testing purposes. With the tree implemented, all the other ones open in a half second or less. (almost isntentaneously for most)

AnotherJake Wrote:2) This operation should not be done on load, only on export from your graphics package plug-in or script, or pre-converted on disk with a little utility app. The game that uses it should simply load the distilled product and not dink around with optimizations on load.

This is for a utility I'm making. It will export in a binary format with the verticies already optimized. I'm definitely not going to use obj for my final format. (text based = bleh; I'm using it to import, though, because it's common and has some nice features)

AnotherJake Wrote:3) I am very surprised that the obj file wasn't optimized to get rid of duplicate vertices to begin with. Not saying they didn't legitimately need to be fixed, but I have my suspicions...

It is optimized, but the problem is it's overly optimized to be used in vertex arrays. It has different indices for the vertex, normal, and texture coordinate. Therefore, I expand all of the face verticies into individual verticies, then compress them back down into what can be used by vertex arrays.

I hope that cleared some things up.