XML- SceneToc vs. dotXSI

March 15th, 2005 by Helge Mathee - Viewed 15093 times -




A lot of companies are going XML lately. As Sony announced at the Alias User Event in Los Angeles (December 3rd, 2004), XML was used for storing a lot of different types of assets for the production of Spiderman and Spiderman 2. Omation spoke up at the Softimage XSI usergroup meeting in Los Angeles (March 10th, 2005), letting the community know how Omation’s pipeline is built on XML to store scene/model information. Just to name two.

XML (Extensible Markup Language) was originally developed for web documents. Designers can create their own tags to store custom data in an organized way to transfer information between applications using XML. Softimage started to support the standard for the ScnToc files, which store a summary of the scene elements in a XML file. With these files, production facilities e. g. are able to switch the reference models used in a scene without opening the scenefile, but by changing ASCII text instead, which is much faster, and even more importantly batchable throughout a whole sequence of scenes.

Furthermore Softimage’s XSI now supports XML as the standard for all layout/view related files. Layouts, views and the new relational views are all based of the ASCII standard. Also this feature enables large production to deal much more flexibly with their interface and generate new relational content on the fly.

dotXSI, Softimage’s own interactive media format, which supports export as ASCII or Binary compressed files, already stores a huge amount of scene-data, including geometry data, animation, camera information, lights etc….constraints, XSI specific operators (ala cluster shape combine), and Animation Mixer relevant data are not supported yet. Softimage took an amazing step by implementing the dotXSI standard, giving the users the ability to store model data in ASCII and enabling them to modify the information externaly of XSI. More than that: There are plugins for the major 3D packages to import/export dotXSI files, Softimage provides an external 3d viewer for dotXSI files, all major conversion programs support dotXSI by now!

Sounding like the ultimate solution for 3D data storage, dotXSI became more and more popular. From a non-linear pipeline point of view though, dotXSI, as well as the binary EMDL and SCN file solution are only partially applicable.

Working with static data such as EMDLs of dotXSI files, the only way to transport information from one model to another one is copying. Practically that means: Open a scene, import two models, copy the UVs from one model to the next, as an example. Considering the size of nowadays productions, the amount of rigs/models they are dealing with and the fact that schedules become tighter and even tighter, the process of copying is simply too slow and inefficient. dotXSI is a great feature of the non-linear package XSI, though the standard itself is totally linear.

That raises the question for alternatives… Here’s the idea (which is already in use here at Omation): Using XML for custom storage of scene/model information, which enables the users to reuse the data without importing the whole dataset.

Description of a cube in XML 1.0 (IE 5.0 or higher)
Description of a camera in XML 1.0 (IE 5.0 or higher)
Description of a light in XML 1.0 (IE 5.0 or higher)

Example:

XML description of a “texture-asset” including Geometry, Clusters, UVs, Rendertrees, Expressions. There could be custom tools which read the whole dataset and generate a model with geometry, applied UVs, materials etc. There can also be a custom tool which only reads the UV information, one which only reads cluster information and applies this information to an existing model in the scene, or even better, a XML description of a model, so you never have to open an EMDL file to actually apply these changes. Furthermore, all these processes would be batchable, so e.g. to get the current UVs on a bunch of models is not a problem, and can be executed by a CRON job on a server on a schedule-basis. The advantage over dotXSI/ScnToc is the non-linearity in a pipeline point of view. Models/Assets/Scenes can be broken up in discreet units for different departments/people to work on, and can be merged together on a per-unit level basis. So e.g. as soon as the texture part of the model is ready all related models get the most current UVs/materials.

This way of dealing with data changes the process of “copying” to a process of “applying” changes in an incremental way. And this is possible on a per model/per sequence/per show basis.

16 Responses to “XML- SceneToc vs. dotXSI”

  1. Francois Lord says:

    This is the kind of workflow that reminds me that the gap between small shops and large ones is widening.

  2. Helge Mathee says:

    oh yeah it definitely is. i think depending on the concept though, you could find something scalable enough for a large company, as well as flexible enough for a small studio…

  3. Jason Dexter says:

    Helge, does Omation only use these XML files through XSI, or is there any cross-app integration? It would be very nice to see such an extensible standard develop between XSI>C4D>Maya etc…

  4. Kai Wolter says:

    Using XML is definetely a very smart way of describing the scene graph as well as the rendertree due to it”s tree like structure etc. Therefore I think ScnToc is just the beginning. I am looking forward to seeing a solution which offers more than just a simple resource lookup/replacement option for the scn file.
    Developing neat and less expensive asset management tools operating on this data becomes possible due to the existing powerfull XML tools (e.g. SAX, XSL-FO) and knowledge. But we should keep in mind that XML adds overhead (time for parsing, filesize) too. So using it for more than storing meta data will lead to new problems to be solved as well.

  5. Stefan Larsson says:

    My first thought when seeing the file examples is that the XML-files seems a bit bloated (low amount of useful information versus decorations). But diskspace is probably not an issue nowadays?

  6. Helge Mathee says:

    Jason:

    We only use XSI, so there is no real need for cross application support… but as it is ascii i don”t see any problem in doing that.

  7. Part of the functionality of many if not most TD tools in my experience is storing and retrieving data.
    Having a consitent format for exporting and importing and storing data is extremely valuable.

    XSI presets are limited to small parts of an object, except in the rendertree where it is allows branch storage but is still limited to direct tree connections, whereas XML allows storage and retrieval of exact parts seperated far in the scene hierarchy, but close in the scene graph, so one just gets the parts required to reassemble a setup of relations, rigs, shader connections to scops etc etc.

    An example would be where one often wants to store the constraints of a character to another as well as the buffer nulls, expression and mixer animation in ONE file to reassemble instead of having to reapply all of them manually or with a custom fuction.

    Jason, I agree, an XSIXMLHoudini etc. format would greatly help.

  8. Helge Mathee says:

    Stefan:

    I agree. The major advantage is just the fact that there are parsers for XML in nearly any language, and that parsing XML doesn”t take too long. Furthermore: It is really easy to transfer XML data structures in a database and back.

  9. luceric says:

    I think the point of the article is confusing.. I believe the underlying subject of the article is referencing, and not really XML itself. There isn”t much avantage to XML compared to the dotXSI format. dotXSI, based on the DirectX format, is extensible, and supports user data and templates, hierachical structured data and can be parsed in various ways, and doesn”t require creating everything. It”s also more compact and can use a binary format — which is necessary for size, but also for numerical accuracy, which is lost when floating point data is written in ASCII. An equivalent structure in XML would be pretty mich the same thing, except with different syntax sugar and one has more choice of parser libraries. XML and the dotXSI language address the same problems in similar ways.

  10. Helge Mathee says:

    luceric:

    What exactly do you mean with “underlying… referencing..” ?

    The major advantage for us at Omation though is the fact that there are parsers for XML already, as there are libraries to convert XML data into a MySQL database and vice versa. Also, a lot of companies start their own ASCII format, which is neat in a way, but I think that it is odd to expect Alias to implement everything for dotXSI, cos they have Maya ASCII already. A standard which is used widely all over the world, and has proven its stability and accuracy is more appropriate, imo.

    If I wrote a 3D package, i wouldn”t expect other 3D software developers to adapt to my own format, but try to come up with a standard supporting all 3D relevant data platform-independently.

    Furthermore XML supports inner-file referencing, which means that you can use something like includes to reference a material onto an object, for example. Other than that there are companies out there specializing in referencing data using XML and databases (AD: http://www.stylusstudio.com), which makes the usage of XML in a production really promising.

  11. Adam Ferrall says:

    XML also presents additional benefits, even when compared to other open ASCII formats. The ability to customize the representation of data, insert specialized tags when required, and store only certain pieces of data is a great advantage. Through XML, you could write tools which contain data to represent certain 3D information, but additionally store dependency or relational information in an easily accessible way. I am not at all saying that XML is the only format which could have these features, only that it can satisfy them easily and there are a number of tools and data structures which allow quick access to XML file structure and information.

  12. luceric says:

    Helge,
    just to clarify XML is not a file format, it is only a way to define a file format, syntax sugar, and it isn”t better than other syntax sugar, in fact it is excessibly verbose and inneficient, a bastard child of the dotCom crash. If two applications had their 3D scene files in XML, they can still be completely incompatible if XML is the only thing they have in common. XML only defines the way the document is tagged and not the meaning of the content, and only loosely providing guidelines for structure. 99% of the work and time of reading and writing a 3D scene file is about the data structure and its interpretation, not the parsing of the ASCII tokens. For parsing ASCII tokens, not only are there libraries for all language, but also higher-level tools such as YACC which aid in understanding the data semantically, at the grammar level.

    To take a Softimage example, dotXSI is based on DirectX”s .X format and the parser library to read the tokens and built a tree for it are found both in the DirectX SDK and the softimage FTK. But there is a problem, while you can parse everything, you need to understand what are the objects defined here. So your parser quicktly becomes much more complex and parsing “parameter1=2.0″ wasn”t the hard part.

    That”s why while there is a low-level API to parser dotXSI file, Softimage also makes a Sematic Layer API, because in reality what a user for dotXSI wants to read is a material, not a list of tokens that eventually, with added programming logic, can be understood to be a material.

    If you only use an XML parser lib, you have no semantic layer. You can now extract all tokens from the file, and even get them in a tree. But you still need to navigate that tree, and understand the data within it with your own logic, which will far exceed the complexity of parsing basic ASCII token.

    With regards to dotXSI vs Maya”s .ma, the comparision cannot be done. Maya ASCII is a pseudo MEL script and dump of the Maya data structure, which means that applications other than Maya cannot hope parse it to re-create the scene, unless they emulate every objects, operators and plug-in in Maya , and keep up with every version of Maya. Same is true for the 3DSmax format or any application ”native” format.

    dotXSI is meant to be publically defined and documented and doesn”t contain contruction history, *by design*, although it can contain user-data (which is necessary for games). It is meant for exchange, just like the filmbox format or the Mirai Games Exchange, the data is simplified to be understood by another application without having to emulate the entier host application.

    dotXSI was never, ever meant to be all-ecompassing 3D format that would be implemented nativly by 3D applications, that”s a solution without a problem. The problem is getting data in and out between completely different apps.

    With my comment about file referencing, what I mean is that what the TD wants to do is replace part of the scene, include another scene, change the connections – or references – of objects. This is what people do when they do when the edit maya ASCII files or the Scene TOC.

    I meant to say that it”s not a factor of XML or not, but rather a question of the way the data is defined in a human-readable file, as a set of references to individual parts, so that a human can to changes in an ASCII file and plug things together differently without breaking anything.

    Whether that human-parsable file is using XML tags or another convention won”t affect the outcome of this and how sucessful it is. In fact, formats that have no hierarchy are much simpler and safer to parse through through batch files or edit by humans while generally XML will require the lib to read the entier document in order to find a token in the middle. The best solutions are the simplest and fastest.

  13. anthonyc says:

    XML v dotXSI

    Parsing:
    ======
    dotXSI: is a very easy parse. (the older versions eg 1.xx are a little messier).

    XML: are an easier parse due to the fact that there are so many XML parsers out there that do the hardest part of the job for you. One parser, MSXML is available on all machines, and since its a COM dll, won”t add to the size of your library/executable.

    Validating
    =======
    XML: you can check whether an XML file has errors by dropping it in IE. IE will tell you exactly where those errors are. If dotXSI were XML, IE wouldn”t be able to tell you whether you typed an XSI specific tag incorrectly, but these are generally easier to pick up that brace, comma, colon problems.

    dotXSI: can”t recall much for dotXSI validation, some messages XSI will give you are quite cryptic (I will use my own parser to check over problematic dotXSI files)

    Browsing
    ======
    dotXSI: easy browing through folding text editors like VC, or http://www.scintilla.com. And also the format itself is *very* easy to read. Perhaps the easiest of any 3D format. There is very little kruft.

    XML: IE presents you with a hierarchical view which allows for much easier browsing of file contents. However 3D XML formats are generally harder to analyse since every number are stuck in their own own element.

    Referencing
    =========
    dotXSI: you cannot.

    XML: you can

  14. RPS says:

    Hi Helge

    I was wondering that can we use XML to create tools in XSI. Like new toolbars something like that. and If yes then can u provide me with some example. I would be really greatful.
    thanx
    RPS