Home

Quick test of snappy compression

2011-04-23

The snapshots from cosmological simulations can get quite large. It's easy to accumulate several terabytes with a 10243 simulation. It would be useful to compress the output if the running time is not significantly affected. This might be a good job for snappy compression.

As a quick test, I compressed a single shard from a small simulation:

Size Time
original 98 MB
snappy 32MB 0.2s
gzip 9.6MB 3.3s
bzip2 5.6MB 28s

In this case, snappy was 17 times faster than gzip but the result was 3.3 times larger. The snappy compression rate is 490 MB/s which is almost twice as large as the value claimed on their site (but of course this strongly depends on the hardware).