FileFormatBlues

Instinctively, XML feels good.

I often think about OpenMPT's future file format but never really get very far code wise. We have to move away from raped XM/IT for so many reasons, and The XML direction should provide value in terms of extensibility, maintainability and portability. I find it very difficult to work on any other major feature (namely better plugin routing + automation, 32bit sample support etc..) if I know the way it's going to be stored will change soon. But every time I set off to work on the XML stuff it I get a bit overwhelmed. Here's why.

Initial complexities and concerns (and learning curve)

I'm no expert, but it seems to me XML is at it's most useful when the schemas closely resemble the actual objects used at run time. That way you 'just' serialise your internal representation of the track to XML to save it, and 'just' deserialise it to load it.

Now, I'm still learning about all this stuff, but from my newbie understanding, there's a bit of an obstacle here: you can't just straight-off serialise/deserialise any object to/from XML at runtime in c++. Object fields can't automagically bind to XML entities, because there's no reflection functionality in C++. You therefore have to rely on code generators to build special XML-serialisable classes, based on your schema definition. One such code generator is this xsd compiler. It supposedly produces C++ code from XSD schemas that work well with the Xerces XML parser/generator. But I'm concerned about:

• the complexities of getting very OO C++ generated code to work alongside MPT's "oldschool" C-ish MFC code, and then maintaining.
• the extent to which this is going to complicate the build process
• how much it will bloat the OpenMPT binaries
• the amount of time and effort it's going to take me to learn to use those tools before we actually see any usable benefit in OpenMPT.

More Tooling to Learn....

We'd really be losing out of we didn't make use of the fact that XML can be designed by drawing pictures. The idea is to generate the XSD schemas mentioned above from UML diagrams. After briefly looking around, it seems the Eclipse EMF plugin along with some UML plugin can do this, though those tools are really intended for Java development. However if we just want the XSD I don't see why this would matter.

Interfacing a Freshly Designed Format with a Rusty Engine.

Supposing all the tooling listed above actually works out, the next concern is what to put in our XML. Currently, when you load a track, MPT processes the IT/XM/whatever byte stream, and fills a horrible monolithic object (class CSoundFile) which contains pretty much everything about that track, plus a whole lot of stuff we shouldn't need to persist to file. We won't really gain that much from building a schema based on that class.

It would make more sense for the schema to represent the way the song object should or will be in the future: nice and modular. But the current monolithic representation is tightly tied into the player engine and the GUI. So initially we'd need to convert between our nice and modular song format to the nasty monolithic internal representation on load and on save… in the hope that bit by bit, the rest of the program will evolve to cope "natively" with our nice and modular format. This would happen mostly as new functionality is added and old bits get refactored in the process.

Now what, Robin, you idiot?

So I think we end up with a set of work items that look something like this:

1. Confirm XSD and Xerces can co-operate with the current OpenMPT codebase, without too much bloat: design and implement simple test case.
2. Learn to use the Eclispe EMF / UML plugins to generate XSD Schemas
3. Confirm the "Eclipse EMF -> XSD -> Generated C++" cycle is viable
4. Dream up a prototype of our XSD schema
5. Write converters for new modular song representation to/from current monolithic internal representation(plus: figure out how to handle the binary (non XML) stuff like sample data; figure out XML+binary packaging (zipped directory structure, probably))
6. Plug ears when everyone complains files now take 10x longer to load and OpenMPT's memory and disk footprint have grown by 500%. :P
7. Slowly tighten format spec and refactor player/gui guts; new data model improves and the rest of OpenMPT adapts to it; converter to/from current monolithic data model slowly becomes obsolete.

Conclusions

When one's intention is to do something well, it's hard not to bite of more than one can chew. There's so much to do, and so much to learn before I can even get started. And, as usual, after that thought has simmered in my mind for a bit, I wonder why I shouldn't just keep hacking the old file format with minor bug fixes and improvement, like I have been doing for the past few years… and quietly wait for everyone to start using Psycle and Renoise. :)

Like I said, I don't like blogs - whenever I read one I'm embarassed on the author's behalf. But I think this has been therapeutical. In a few hours time I'll need to go to work, and when I've slept through that maybe I can start solidifying the work items… or realise they are bollocks.