XSIBlog

A few years ago I did some work to make the .pic parser blazingly fast. One day, I noticed that the load time for an image sequence had doubled.

That’’s a big difference! Waiting 6 seconds instead of 3 for a sequence to preload in the flipbook can have big impact on how you work, especially in preview workflows. These sort of performance shifts can also make the difference between a compositor being disk-bound instead of CPU-bound.

After some panic and investigation, I discovered that the source of the problem was that I had moved all my test image files to another drive and in the process the image files got fragmented. Since the .pic parser had been tuned using memory mapped files and was very fast to begin with, it was the most hit. Other parsers that had a more lazy implementation, for example relying on reading the images line-by-line, showed little difference. In other words, the slow image parsers remained slow.

One tool I used to diagnost the problem was Contig from SysInternals. This utility allows viewing how much a file is fragmented, and allows defragmenting individual files. (This tool does not work on network disks or ‘’subst” drive letters) You can use to tell if fragmentation is a problem or not, without having to defragment your entier drive.

When a software reads a file, the disk starts spinning and the OS is filling the disk cache with the bits that the software will probably soon need. When the file is fragemented, the disk will have to jump around to find the different bits for each fragment, and the performance goes down dramatically as all the time is wasted realigning the heads instead of reading data. Unfragmented video storage is essential to get real-time playback NLE workstations such as Avid, and libraries like Quicktime make some effort to minimize fragementation. Very little else does, however. Most applications write files chunk-by-chunk or line-by-line, therefore the OS has no idea how long the file will be and this leads to the files being fragmented. For example, the output of 3D renders, especially if it goes in one gigantic directory, is generally very fragmented. Which means it”ll be long to load these files into a flipbook or a compositor.

It’’s possible for a software to give hints to the OS to minimize fragmentation when writing a file. The trick is create a new file, then seek at the future end and write a byte. This tells the OS how large the file will be before anything gets written, and in my experience leads the OS to take better allocation decisions.

If you write a lot of files on your workstation, whether with rendering or code compilation, I believe it is necessary to defrag frequently to maintain your system’’s performance, especially on NTFS. Personally I use Diskeeper, which is fast and effective. It has a Set It And Forget It that will make defrags automatically, and only when it is necessary, on a given schedule. So you could set it at 4AM every day to run for 10 minutes, for example. The only sad part of this for them is that it is easy to forget that you have this software doing all this optimizing of your system during the night. The defrag that’’s built-in Windows is based on Diskeeper, and it’’s better than nothing but it’’s slow.

This entry was posted on Monday, November 14th, 2005 at 1:37 pm and is filed under General. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.