Wishful 3D file format

Member
Posts: 131
Joined: 2004.10
Post: #46
AnotherJake Wrote:For example, it makes more sense to me to have a single tag saying how many vertices to follow instead of prefixing every line with a "v" for vertex. Two reasons: 1) I've said this before and I'll say it again, it is often easier from a parsing standpoint to have a single tag coded to say that "there will be eight vertices following this tag" so that memory can be allocated right then and there without having to calculate from some other dependent variable. Or even worse, having to actually parse the file once extra just to count! 2) for something like 200 vertices, having a tag like # Vertex 200 takes up only twelve 12 bytes on disk whereas prefixing every vertex line with a v would take up 200 bytes on disk, plus extra work to lex over the v. It just seems unnecessary to me to have v, or whatever, on every line when I already know darn well what it is I'm parsing.

Can't dispute that. I always wondered why they didn't do that.


AnotherJake Wrote:That reminds me. Wouldn't it be beneficial for file size to *not* use 1.00000 to represent 1.0? Why not just stop listing zeros when the first least significant zero digit is encountered? You wouldn't say 00034.34 would you? I often see something like 52.23200000 in an ASCII file. That should just be 52.232 IMHO. I suppose the argument could be made that it might make the alignment easier on the eyes, but... Does it really? I hardly ever see spacing in the file to accomodate a "-" sign and that screws alignment up anyway. That's not to say that extra zeros shouldn't be allowed, but I would say that it should be encouraged to leave them out.

Heh. I thought the same at work but apparently some older fortran code may 'require' fixed width fields. I'm not saying this is the case in this situation but I hit this case at work and it annoyed me to no end that an XYZ format had to have it's fields fit into 16 columns. But in this case I think it's mainly lazy/don't care what gets tossed out of the exporter that made the file.

In fact it should just send out 1 not 1.0.
Quote this message in a reply
Moderator
Posts: 3,579
Joined: 2003.06
Post: #47
Zekaric Wrote:In fact it should just send out 1 not 1.0.
That'd be fine with me!
Quote this message in a reply
Moderator
Posts: 771
Joined: 2003.04
Post: #48
AnotherJake Wrote:[...]For example, it makes more sense to me to have a single tag saying how many vertices to follow instead of prefixing every line with a "v" for vertex. Two reasons: 1) I've said this before and I'll say it again, it is often easier from a parsing standpoint to have a single tag coded to say that "there will be eight vertices following this tag" so that memory can be allocated right then and there without having to calculate from some other dependent variable. Or even worse, having to actually parse the file once extra just to count! 2) for something like 200 vertices, having a tag like # Vertex 200 takes up only twelve 12 bytes on disk whereas prefixing every vertex line with a v would take up 200 bytes on disk, plus extra work to lex over the v. It just seems unnecessary to me to have v, or whatever, on every line when I already know darn well what it is I'm parsing.

I think that's addressed in the format I posted previously, and although it's binary, an equivalent ASCII version could be made, but I still think that the "text editor" approach for building models is not good for anything more complex than a cube, so, if you are going to use a 3D app for making the model in the first place, why make the format "human readable" in the first place?
I can see the benefit of storing things like extra animation data (see previous post), as text, since it would make sense to modify that by hand (see my previous post), but for the main file it just seems like a waste of space & loading time.

AnotherJake Wrote:That reminds me. Wouldn't it be beneficial for file size to *not* use 1.00000 to represent 1.0? Why not just stop listing zeros when the first least significant zero digit is encountered? You wouldn't say 00034.34 would you? I often see something like 52.23200000 in an ASCII file. That should just be 52.232 IMHO. I suppose the argument could be made that it might make the alignment easier on the eyes, but... Does it really? I hardly ever see spacing in the file to accomodate a "-" sign and that screws alignment up anyway. That's not to say that extra zeros shouldn't be allowed, but I would say that it should be encouraged to leave them out.

Using a binary format, each float would take only four "chars" Grin

AnotherJake Wrote:Here are a couple of ideas for a vertex tag with one example vertex line following each:
Code:
# Vertex 128
1.0, 2.23, 0.0

<Vertex 200>
1.0, 2.23, 0.0

V 320
1.0, 2.23, 0.0

Vertex 150
1.0, 2.23, 0.0
The third and fourth examples would make the parser assume that a non-numeric character denotes the beginning of a tag. Using something like a # or < makes parsing much more efficient however since it is immediately obvious.

Actually, considering that the parser would have to parse text strings like "1.9045864" into floats and detect end of lines, you are already being less efficient than an equivalent binary format (in a binary format, at most you'll need to swap bytes on intel-like hardware, extremely easy & fast)

Again, I don't see the advantage of being able to "read" a 10000 vertex file as text. Huh
Quote this message in a reply
Moderator
Posts: 3,579
Joined: 2003.06
Post: #49
I completely agree that a binary format has every performance/storage advantage over text, I would be a fool to suggest otherwise. All of *my* files are binary as well, for the reasons you just outlined. However, I don't think that a binary format to begin our design with is very intuitive to work with in a group setting, and especially with beginners. Plus, there shouldn't be any difference in the end since a binary version should easily be made when the design is more mature.

On your format, shouldn't you be using unsigned int or unsigned short in your header instead?
Quote this message in a reply
Moderator
Posts: 771
Joined: 2003.04
Post: #50
AnotherJake Wrote:On your format, shouldn't you be using unsigned int or unsigned short in your header instead?

Definitely, I just use int for everything Rasp ! But yeah, they should be unsigned.
Quote this message in a reply
Moderator
Posts: 771
Joined: 2003.04
Post: #51
OK, how about making a text version of the format I outlined previously, and adding the support for sub-objects/multiple textures, and anything else that may seem indispensable?
Ideas on how to implement sub-objects?
Quote this message in a reply
Member
Posts: 715
Joined: 2003.04
Post: #52
One texture per file is terrible, you should support materials and better still shaders.

Leave all the "binary" discussion until the end, since each dev could convert the ascii to a specific binary format for their own games, adding some security tags, should they not want their content available to everyone and anyone.

I thought Okugai had used an .md2 loader? thats whats in the package.

By the way here's [url= http://www.igame3d.com/WTFPreviews1to99/ ]the first 99 models that I converted to .WTF[/url]
(I have over 600 in .mesh, about 7600 in .lwo, .c4d .3ds, .obj .uNameit)

Made the convertor interface, did all the imports, textured, exported and made
the screenshots all in one night. I think it was 4 hours of converting, the 87,006 poly aztec city really gave me trouble, but I was dertermined to export it anyway!
Not all great models, I was just going down my list of 600 in alpha-numeric order.
Quote this message in a reply
Moderator
Posts: 771
Joined: 2003.04
Post: #53
igame3d Wrote:One texture per file is terrible, you should support materials and better still shaders.

Agreed, as I said, I wrote it in a couple of days, with just the basic features I needed.

igame3d Wrote:Leave all the "binary" discussion until the end, since each dev could convert the ascii to a specific binary format for their own games, adding some security tags, should they not want their content available to everyone and anyone.

Already did Wink

igame3d Wrote:I thought Okugai had used an .md2 loader? thats whats in the package.

It uses .md2 for the characters, .jig (the format I described) for all the other models (including the title).
Quote this message in a reply
⌘-R in Chief
Posts: 1,265
Joined: 2002.05
Post: #54
Zekaric Wrote:One major bonus with an ASCII option is if the format evolves, older programs can ignore what they don't understand. This is easier to do in ASCII than it is in Binary. However I think if the format is crafted in a clever way this may not be at all an issue.

It isn't an issue at all if you use key-value coding methods. If the parser comes across a key (ascii or binary, it doesn't matter), it skips until the next key. With an ASCII format, keys could easily be differentiated from data using start/end lines, and in binary you can use a simple, "key, length of data, data" format. Not an optimum space saver, but certainly easy to parse and very very extensible.

If someone actually came up with what actually needs to be stored in the file (not a format, just a list of what data), it'd be a piece of cake to come up with a decent format, and just a couple hours of time to create a parser.

Can someone make such a list? I can start with simple vertex and texture coordinate stuff with frames, but beyond that, it's out of my current realm of knowledge.

I'm totally dedicated to the idea and am willing to put the time into it. I just need a bit of guidance.


A couple other thoughts: the idea of XML was neat, but it just isn't practical. The need for libraries and stuff makes it too complicated and scary for anyone who hasn't done it.
Quote this message in a reply
Member
Posts: 446
Joined: 2002.09
Post: #55
I'm a little late to the party but I've been using a custom tagged-chunk 3D format for months now that encompasses many of the ideas in this thread and allows for both binary and plain text files.

You can read some specs here, and see an example here. This doesn't explain everything, but the format is flexible and extensible, and very readable in text format.

I have cross-platform loading & saving routines, but they're written in REALbasic, so poo-poo that fact if you must. Writing binary I/O in C should be easy enough, but parsing text in C is a pain in the arse...
Quote this message in a reply
DoG
Moderator
Posts: 869
Joined: 2003.01
Post: #56
FreakSoftware Wrote:It isn't an issue at all if you use key-value coding methods. If the parser comes across a key (ascii or binary, it doesn't matter), it skips until the next key. With an ASCII format, keys could easily be differentiated from data using start/end lines, and in binary you can use a simple, "key, length of data, data" format. Not an optimum space saver, but certainly easy to parse and very very extensible.

If someone actually came up with what actually needs to be stored in the file (not a format, just a list of what data), it'd be a piece of cake to come up with a decent format, and just a couple hours of time to create a parser.

Can someone make such a list? I can start with simple vertex and texture coordinate stuff with frames, but beyond that, it's out of my current realm of knowledge.
...

Hey, I am positively surprised by you, Seth. Kudos for the good thinking. Ninja

Now, to get to the gist of it, the low-level format really doesn't matter much. Binary, text, xml, all the same. That is the reason why I asked for what features people want in the file format, not if they like square or curly brackets better.

I will start a new thread on what I am doing, put up some graphs etc. Much talk in this thread has been on how to deal with low level issues, but I am more interested in how to store a scene-graph and how to fit in different object properties, materials, transformations, animation, and so on.
Quote this message in a reply
⌘-R in Chief
Posts: 1,265
Joined: 2002.05
Post: #57
Frank C. Wrote:I have cross-platform loading & saving routines, but they're written in REALbasic, so poo-poo that fact if you must. Writing binary I/O in C should be easy enough, but parsing text in C is a pain in the arse...

Yeah, text parsing isn't very fun to do. In Cocoa it's made a bit easier because of the NSString class, but in general I think binary files are much easier to work with on the programming side.
Quote this message in a reply
⌘-R in Chief
Posts: 1,265
Joined: 2002.05
Post: #58
PowerMacX Wrote:Just as an example, this is the (binary) format I made for Okugai:

That's very similar to the md2 file format. It's not very flexible, but adding some other bytes to flag off sections would. A simple v1 format could easily contain just this kind of information.


Quote:I made an OBJ importer, that can take a series of OBJ and convert it to this format, making each OBJ a key frame.

That's a cool idea. How do vertices between the obj files stay in the same order if you're doing interpolation?
Quote this message in a reply
Moderator
Posts: 771
Joined: 2003.04
Post: #59
FreakSoftware Wrote:How do vertices between the obj files stay in the same order if you're doing interpolation?

Luck Rasp ! I used Wings 3D and, as long as you don't add or remove a vertex to your model, every time you save the model the vertices stay in the same order, and that's what my basic importer relied on. So you can twist, deform, translate, rotate, etc. each the model (or individual faces/vertices) to create each individual keyframe, as long as you don't add or remove vertices.

As I noted, the "format" I suggested is basic, but something like that could be used for sub-objects, so each sub-object could have its own texture and individual transformation. Material info (diffuse, specular, etc) can also be added as some have suggested.

So, here is my v0.1 list of elements the format should support:
  • Vertex (not like md5, REAL vertex positions)
  • Normals
  • Faces (I like triangles-only for easy rendering, but feel free to suggest alternatives)
  • Texture coordinates
  • Materials (or shaders)
  • Textures
  • Subobjects
  • Bones, as long as you dont have to use them (for instance, if you want to model non-character objects)


About how some of this items could be represented: Having each frame as a "modified version" of the "basic object", by having the same number of vertex is a limitation but really facilitates animation. Another limitation in my original format is that the texture coords are shared by all frames, but this can be left as an option, to allow texture animation.

Different lists of items? Keep posting Smile
Once the list is agreed on, implementation is easy, whether ASCII or binary.
Quote this message in a reply
Moderator
Posts: 771
Joined: 2003.04
Post: #60
Frank C. Wrote:I'm a little late to the party but I've been using a custom tagged-chunk 3D format for months now that encompasses many of the ideas in this thread and allows for both binary and plain text files.

You can read some specs here, and see an example here. This doesn't explain everything, but the format is flexible and extensible, and very readable in text format.

From the specs:
Quote:- Flexible -
Loading and saving is automatic, even for application-specific data, and the
classes make no assumptions about how an application will use the data found
within the file. One application may render a model with Quesa while another
renders with OpenGL, animations may be keyframed, skeletal, or nonexistent.
In fact, an E3M file need not contain any 3D data at all - though that kinda
defeats the purpose!


I agree that the format should be extensible, but "making no assumptions" about animations doesn't sound that good. If this format should become widely used, it has to have at least one standard for key framed animation and one for bones, otherwise general exporters & importers for animations would be impossible.
Quote this message in a reply
Post Reply 

Possibly Related Threads...
Thread: Author Replies: Views: Last Post
  what's the most suitable graphic file format when deadling with OpenGL? anthony 8 4,844 Feb 16, 2007 06:25 PM
Last Post: PowerMacX
  Reverse Engineering 3d File Format nabobnick 1 3,021 Aug 13, 2004 12:11 PM
Last Post: nabobnick