A feed handler writes 20,000 messages per second across 8,000 symbols for a 6.5 hour session, 48 bytes per row after encoding. Size one trading day on disk in gigabytes, then state the partition key and sort order that make a query for one symbol over one hour read the least data.

A feed handler writes 20,000 messages per second across 8,000 symbols for a 6.5 hour session, 48 bytes per row after encoding. Size one trading day on disk in gigabytes, then state the partition key and sort order that make a query for one symbol over one hour read the least data.

Approach: Multiply the message rate by the session length for the row count, then multiply by the row width. For the layout, ask what a reader must be able to skip without decoding it.

22.5. One session is 22.5 GB on disk. The row count is 20000 * 6.5 * 3600, which is 468,000,000 rows, and 468e6 * 48 bytes gives 22.46 GB, so 252 sessions a year is about 5.7 TB. Partition by session date, because every research query is bounded in time and a date partition is the only pruning that costs nothing to maintain. Inside the partition sort by (symbol, timestamp), so one symbol occupies a contiguous run of row groups. A columnar reader then uses the row group statistics on symbol and timestamp to skip every group whose min and max exclude the requested symbol, so a one hour single symbol query touches a handful of groups instead of the whole day. Sorting by timestamp first spreads that symbol across every group, and the min and max on symbol then cover most of the alphabet in every group, which defeats the skip entirely. Row group size matters too: at 128 MB a day is about 175 groups, which is coarse for 8,000 symbols, so 16 MB groups trade a larger footer for a much finer skip.

Follow-up: How does the layout change if the same store must also answer a query for every symbol in a single millisecond?

Key concepts: partition by date, sort by symbol, row group statistics, predicate pruning.