Data Quality Checks Every Research Pipeline Needs

Monotonic timestamps, duplicate sequence numbers and price band checks catch the failures behind a wrong backtest.

Every research pipeline handling market data needs, at minimum, five checks run before any data reaches a backtest: monotonic timestamps within a symbol and a session, no duplicate message sequence numbers, prices held inside a rolling band around a reference value, volume checked for an unexplained spike at a session boundary, and price discontinuities reconciled against known corporate actions such as splits. Each check exists because it catches a specific, observed failure mode in real feeds and vendor files, and a pipeline that skips any one of them can still run without error while quietly producing a wrong research result.

Within a single symbol and a single trading session, timestamps should strictly increase, or at minimum never decrease, from one message to the next, and a violation of this usually points to one of a small number of causes: messages from two different feed lines being merged without proper arbitration, a clock adjustment mid-session on the source system, or a vendor file that has been reordered during transfer or storage. A backtest built on out-of-order timestamps can compute a return using a price that has not actually happened yet relative to the point being evaluated, which produces a result that looks correct until the strategy trades on real, ordered data and the result does not…

The check itself is simple to implement: compare each timestamp to the previous one within the same symbol and session, and flag any case where the new value is not greater than or equal to the prior one. The value of the check is not in the implementation but in treating a violation as a reason to quarantine the affected batch rather than sorting the data and moving on silently, since sorting hides the underlying problem rather than explaining it.

Exchange feeds and many vendor formats carry an explicit sequence number so that a consumer can detect a gap or a repeat in the message stream. A duplicate sequence number usually means the same underlying event was recorded twice, often because a pipeline reprocessed a file without properly clearing the prior write, or because two redundant feed lines were merged without deduplication logic.

A duplicate trade record, left uncaught, inflates volume and can double-count a price move in any statistic computed from a running sum, which is a particularly damaging failure because the resulting number is wrong in a way that still looks like a plausible trading day rather than an obvious error.