A trade price column holds 400 million prices for one symbol over a year, stored as 8 byte doubles. Explain why converting to integer ticks with delta encoding compresses far better than a general purpose compressor applied to the raw doubles, and state the bytes per price you would expect.

A trade price column holds 400 million prices for one symbol over a year, stored as 8 byte doubles. Explain why converting to integer ticks with delta encoding compresses far better than a general purpose compressor applied to the raw doubles, and state the bytes per price you would expect.

Approach: Ask what the bit pattern of a double looks like to a byte oriented compressor, then transform the column so that consecutive values differ in a small number of bits.

0.5 bytes per price. A double stores a mantissa whose low bits are close to random, so a byte oriented compressor finds no repeated substrings and returns roughly 1.1x. Multiply the price by 100 and store fixed point integers instead, then take the delta against the previous row in the same column. Consecutive trades in one symbol move a few ticks, so the delta column is a low entropy column of small signed integers, typically inside -8 to 8. Bit packing writes those in 4 bits, and a frame of reference codec storing one base per block of 128 values keeps the occasional jump cheap. 4 bits is 0.5 bytes, so 400m prices fall from 3.2 GB to about 200 MB. Two conditions are required. The column must be sorted by symbol then time, because a delta taken across interleaved symbols is a jump rather than a tick. And the scale must be exact, so an instrument quoted in thousandths needs a factor of 1000 chosen per instrument, otherwise the cast loses a tick and the delta encoding carries that error forward for every later row.

Follow-up: What breaks in the delta encoding when the instrument changes tick size in the middle of the year?

Key concepts: delta encoding, bit packing, fixed point integers, low entropy column.