Parquet, Arrow and Columnar Storage for Market Data

Parquet on disk and Arrow in memory remove the serialization cost sitting between every step of a research pipeline.

Parquet is a file format for storing columnar data on disk, and Arrow is an in-memory columnar format that multiple tools can share without copying or reserializing data between them, and the pairing matters for quant research because it lets a pipeline read a Parquet file from disk directly into the same memory layout that a query engine, a machine learning library, or a plotting tool can operate on without an intermediate conversion step. On a dataset the size of a year of United States equity tick data, avoiding an unnecessary copy or reserialization at each handoff between tools is often the difference between a research query that returns promptly and one that spends most of its time…

A row-oriented format stores all the fields of one record together, so reading a single column, such as the trade price, still means reading every other field of every row along the way, and a query that only needs two of twenty columns pays the cost of the other eighteen anyway. A columnar format stores each column contiguously, so a query naming a small subset of columns reads only those columns from disk, which for a wide table with dozens of fields can reduce the actual bytes read by an order of magnitude.

Columnar storage also compresses better than row storage for financial data specifically, because values within a single column, such as a sequence of prices for one symbol, tend to be similar to one another and repeat patterns that a compression algorithm can exploit, whereas a row mixes a price, a size, a timestamp and a symbol string together, which are different enough in nature that compressing them as a unit works less well.

Parquet is the columnar file format most widely adopted in quant research and in the broader data engineering field, storing each column with a compact, typed encoding, per-column compression, and block-level statistics such as the minimum and maximum value in each block that let a query engine skip blocks that cannot contain a match. A file is organized into row groups, each holding a chunk of rows across all columns, and metadata at the end of the file describes the schema and the location of each column's data within each row group.

Because the schema is embedded in the file itself, a Parquet file is self-describing: a reader does not need a separate schema definition to know that a column is a 32-bit float or a timestamp, which removes an entire category of mismatch bugs that occur when a CSV's inferred types drift between two different loads of what is supposed to be the same data.