You must store 2 TB per year of trade and quote data. Research runs whole day scans over five years. A monitoring dashboard needs the last 10 minutes for 50 symbols within one second. Argue for parquet on object storage, a time series database, or both, and state the cost of the option you reject.

You must store 2 TB per year of trade and quote data. Research runs whole day scans over five years. A monitoring dashboard needs the last 10 minutes for 50 symbols within one second. Argue for parquet on object storage, a time series database, or both, and state the cost of the option you reject.

Approach: Separate the two access patterns by read shape and latency budget, then ask which storage engine each one is built for and what a single engine would have to give up.

Run both: parquet on object storage for the five year history, and a time series database holding only the last few days as the hot tier. The research read is a full scan of 400 GB per year with no point lookup, and object storage gives high aggregate scan throughput per reader at a storage price around ten times below block storage, so parquet with date partitions and column pruning is the right cold tier. The dashboard read is 50 short ranges with a one second budget, and object storage answers that badly because each request pays 20 to 100 ms before the first byte, and the last 10 minutes sit in files still being written, which is the small file problem: thousands of tiny objects that list slowly and compress badly. A time series database holds that window in memory and on local disk and answers in single digit milliseconds. Rejecting the hot tier costs the dashboard its latency and forces a compaction job to rewrite tiny files continuously. Rejecting parquet costs the storage bill and locks five years of history into one engine's format. The boundary is a retention setting plus a scheduled job that seals a completed day into parquet.

Follow-up: How do you stop a query from double counting a day that exists in both tiers during the seal?

Key concepts: object storage scan throughput, hot tier, cold tier, small file problem.