On avoiding TIFF files

TIFF files.

TIFF (Tagged Image File Format) files are used by several notable two-photon imaging software packages to store frame data. TIFF files appeared in the mid-1980s and the latest update, TIFF 6.0 was issued in 1992. Needless to say, TIFF files are woefully outdated - one of the biggest limitations of TIFF file is their size, which cannot be larger than 4 GB.

With conventional scanners, when imaging 512 x 512 frames at 2 fps on two channels, a TIFF file would be full in about 34 min. As I mentioned earlier, this would happen in a little more than 2 minutes in resonant scanning. Writing files in the BigTIFF format could bypass the 4 GB size limitation, but this new file format is not widespread.

Another disadvantage of TIFF (and BigTIFF) files is their internal organization (or lack of). Frame data can be placed anywhere in the file, thus defeating most streaming strategies. In particular, TIFF readers cannot expect the n-th frame to be located at a predictable location related to the frame index - the entire file has to be read until that frame is found, which is arguably a performance problem with large files.

Lastly, the collection of image tags is not extensible so important two-photon scanning information has to be placed arbitrarily in the image description section of the file.

For those who are accustomed to analyzing data in TIFF files, MView provides MCS to TIFF conversion, either for individual files or for batches of files in the same folder.

Structured Storage files.

MCS uses an advanced file format that completely overcome the shortcomings of TIFF files. Firstly, MCS files are based on the Structured Storage technology available under Windows. Structured storage files are like miniature file systems that contain arbitrary numbers of folders called storages which in turn can hold either other storages or internal files called streams.

An immediate advantage of such files is the ability to store analog data (e.g., electrophysiology) in a separate stream alongside imaging data. Individual streams can be as large as 16 TB, so there is little concern that Structured Storage files will run out of space in the foreseeable future. Writing or reading data to or from  the data file is straightforward, thus efficient, during streaming operations. MView has no difficulty opening and reviewing data files larger than 100 GB. In principle you can think of adding all kinds of parallel streams of data, for instance visual stimuli patterns without disturbing the structure of the file.

Similarly, file properties are stored in an independent stream. MScan writes standard file property / value pairs (e.g., microscope lens resolution, stimulation parameters, etc…) but you can add also your own property names and values to the file (e.g., "Animal Line" = "Wfs1-Tg2-CreERT2 : Ai38").

During analysis by either MView or MCSX, file properties can be queried programmatically for property values. You can also visualize the property name / value list of any file without opening it by right-clicking on the file name in any Explorer window.

Structured Storage is a powerful technology used by many professional Windows programs including Microsoft Office, MSSQL, Dassault Systemes SolidWorks and other CAD programs. By using Structured Storage, MCS provides two-photon experimenters with a whole new level of capabilities during imaging and data analysis.