Tunnel Survex reader

From Fs_wiki

Why is there a svx reader in Tunnel?

The Survex cave surveying system is a set of distinct programs for processing, viewing, and outputting the cave data in various formats. The main Survex Data File Format, in the form of .svx files, is what you manually type in. The Survex program cavern processes this into a .3d file which contains the positions of all the stations after the loops have been closed. All other Survex programs read .3d files and either view them, or outputs them in another form.

Why doesn't Tunnel read .3d files like everything else, rather than attempting to parse the highly complex, unprocessed, and command over-loaded .svx files, especially as it doesn't even do it properly?

This question has been subject to a lot of debate, and you need to understand what's going on "under the hood" of the .3d file in terms of what it contains, what it could contain, why it contains some things and not others, and the fact that the needs of Tunnel have no bearing on what it will contain, before considering it as a viable option for building Tunnel data onto.

What could the .3d file contain?

The Survex program 3dtopos converts a .3d file into a .pos file, which is a simple ascii file that lists all the XYZ coordinates followed by the survey station on each line.

( Easting, Northing, Altitude )
(   43.54,   307.09,   -73.80 ) erwang.shuang.16
(   25.63,   313.17,   -13.78 ) erwang.shuang.connection.6
(   28.04,   316.75,   -20.69 ) erwang.shuang.connection.7

These coordinates are the result of the loop closure and error distribution processing done by Survex, which can take several minutes to calculate on a large and complex cave system.

Tunnel does not implement any loop closure methods, and merely works out a rough approximation of the station locations using triganometry along each leg and ignores the loops. Tunnel can, however, read a .pos file that has been put into the appropriate directory and interpret the stations into their correct Survex calculated locations.

Clearly, a .3d file can contain nothing more than the data in the original .svx files and its associated .pos file. Subject to being able to read the byzantine but at least readable .svx file format (once you've got so far there's too much to throw away), and the .pos file (which is easy) this is by far the safest option, because (a) the svx files are unlikely to be radically overhauled when there is so much manually entered data in this form, and (b) you are not limited by any design decision that has been made with the 3d file.

What does the .3d file contain and why?

The purpose of the .3d file is for visualization of cave data. It's a binary format so that the files are small and quick to load into whatever viewer that you are using. They could have used the (now defunct) VRML format, which would have made it possible to use many different viewers and even embed surveys into a web-page, but the intuitive requirements of speed and efficiency were thought to be more important than such features.

While cave surveys look large and complex, their file sizes are small by comparison to most CAD files, since in CAD there are 2D surfaces to specify, while cave surveys are made out of one dimensional lines. The difference in loading times of an general purpose ascii file format, and a specialized binary format are imperceptable on all computers that are powerful enough to animate the visualization, and the on-board RAM requirements within the computer are exactly the same no matter what form the file is in, so in terms of memory efficiency the only thing that's relevant is the size on the disk.

Because .3d files are supposed to be small and efficient, they contain as little of the data from the .svx files as possible; just enough to run the features in aven, the visualization program. As the Survex developers added new features into aven, they required more of the data to be encoded in the .3d files, which is now on its fourth incompatible version and is beginning to resemble the TIFF (Thousands of Incompatible File Formats) Format.

There is no documentation of the .3d format on the Survex web-page, although some people have found it easy to reverse engineer it.

What does Tunnel need a .3d file to contain?

Apart from all the huge numbers of calibrations and wacky survey leg formats that Survex can support, and which its unimportant to get right if you read the positions from the .pos file, the most important data is the way survey stations are linked.

In cave data prepared for Survex, surveys are linked at their stations by the *equate command. This takes the names of two or more survey stations and states that they refer to the same location. The .3d file is written after Survex has applied these equates and you lose all but one of these alternate names.

This doesn't necessarily matter for visualization, because you have either calculated a 3d file specifically for one survey trip, in which case there are no equates, or you have a 3d file for the whole cave and these minor details are unimportant.

What it does mean is you can't extract a single survey trip out of a larger 3d file and expect to see all the stations which were written down on the paper. Furthermore, if you were to base an entire survey around a 3d file in one year, and then you extended the cave the next year, your new 3d file might not be compatible since it would not necessarily contain all the stations from the previous year -- some of them will have been renamed by the new equates, and there would be no way to tell what the missing stations had been renamed to.

A full set of alternative names for each survey station is essential to handle these cases of overlapping surveys, but since would consume a few more micromegabytes of disk space and slow down the loading by a few miliseconds, it won't be considered.

History

The above issues were not known when Tunnel started out and began by reading svx files. The original version generated editable cross-sections around the stations derived from the informal Left/Right/Up/Down data that was encoded in the CUCC data files. This information certainly wasn't going to be in the 3d file format, and there was no point in waiting for it.

Previous (and possibly the current) versions of the 3d format also don't take account of the *title information, which is used as the basis of the Sketch Subsets, or the *date values, which has been used to animate the size of the cave over time, or the original survey measurements, which can be used to colour how much the legs have been distorted by the loop closure process.

See also Problems with the Tunnel Survex reader.