Building a Tick Data Pipeline

A tick data pipeline needs ingestion, normalization, validation and partitioned storage run in that exact order.

A tick data pipeline for quant research has four stages that have to run in order: ingest the raw messages from the exchange feed or a vendor file, normalize them into a consistent schema, validate them against a set of data quality checks, and write the result to partitioned storage that a backtest can query efficiently. Skipping or rushing any one of these stages is the most common reason a research result later turns out to be wrong rather than merely slow, because a pipeline that runs without errors can still silently pass along corrupted or misaligned data.

Raw tick data arrives either from a direct exchange feed, encoded in a binary protocol specific to that venue, or from a vendor file that has already done some of the decoding, typically as a compressed text or binary format organized by date and by symbol. A pipeline built for research rather than live trading almost always starts from vendor files, since decoding a raw exchange feed from scratch is a substantial engineering project on its own that most research teams do not need to repeat.

The ingestion step should do the minimum necessary work: read the raw records, keep the original fields, and avoid any transformation that could lose information before the data has even been checked for correctness. A common mistake is combining ingestion and normalization into a single pass, which makes it much harder to isolate whether a bug in the pipeline originated in the raw data or in a transformation applied to it.

Different venues and different vendor formats encode the same concept, such as a trade condition code or a symbol identifier, differently, and a normalization stage maps all of these onto one consistent schema that the rest of the pipeline can rely on without special-casing the source. This stage also handles corporate actions: a stock split changes the meaningful comparison between a price on one day and a price on another, and a pipeline that does not adjust for splits will show a price series with a discontinuity that has nothing to do with the market and everything to do with a share count change.

Timestamps deserve particular care during normalization, since a feed can report time in the exchange's local timezone, in UTC, or in a vendor-specific convention, and mixing these without converting to one consistent representation is a common source of subtle misalignment between series that are supposed to share a clock.