The snapshots from cosmological simulations can get quite large. It's easy to accumulate several terabytes with a 10243 simulation. It would be useful to compress the output if the running time is not significantly affected. This might be a good job for snappy compression.
As a quick test, I compressed a single shard from a small simulation:
Size | Time | |
original | 98 MB | |
snappy | 32MB | 0.2s |
gzip | 9.6MB | 3.3s |
bzip2 | 5.6MB | 28s |
In this case, snappy was 17 times faster than gzip but the result was 3.3 times larger. The snappy compression rate is 490 MB/s which is almost twice as large as the value claimed on their site (but of course this strongly depends on the hardware).