Time Series Storage Formats for Quant Research

Partitioned Parquet files beat a single CSV or a row-oriented table for querying a year of tick data efficiently.

The storage format a research pipeline chooses for time series data determines whether a year of history loads in seconds or minutes, and the short answer for most quant research is a columnar, compressed format such as Parquet organized in files partitioned by date, rather than a single large CSV or a row-oriented database table queried directly. The choice matters most at the scale a single United States equity trading day actually produces: on the order of tens of gigabytes of raw message data once quotes, trades, and order book updates are all included, which makes a naive format the difference between a query that returns in seconds and one that stalls a whole research session.

A CSV file stores every column of every row together, which means reading a single column, such as the trade price, still requires reading every other column on every row just to find it, and parsing text into numbers on every load, which is slow and repeats work that never changes between reads. A single day of message data for an active symbol easily reaches hundreds of megabytes as a CSV, and a year of history across a few hundred symbols reaches the scale where loading the whole thing into memory as a single pandas frame simply fails on an ordinary research machine.

A row-oriented database table has a related problem for research workloads specifically: research queries typically ask for one or two columns, such as price and size, across a long time range and many symbols, which is exactly the access pattern a row store handles worst, since it still touches every column of every row that falls inside the requested range even though most of those columns are never read.

A columnar format stores each column contiguously on disk rather than interleaving all the columns of a row together, which means a query asking only for the trade price and the timestamp reads only those two columns, skipping every other field entirely. Parquet is the columnar format most widely used in quant research, and it stores each column's values with a compact, typed encoding rather than as parsed text, which both shrinks the file and speeds up the read, a topic covered in more depth in Parquet, Arrow and Columnar Storage for Market Data.

Parquet also stores per-column statistics, such as the minimum and maximum value in each block, which lets a query engine skip entire blocks of data that cannot possibly contain a match before reading them at all. A query filtering for a specific price range on a specific day can skip most of a file this way, turning a scan of the full dataset into a scan of only the relevant fraction.