A streaming job has written hourly PnL aggregates for six months and a bug is found in the calculation. Backfill six months while the live job keeps running, with no double counting and no gap. Describe the procedure and the exact cutover step.

A streaming job has written hourly PnL aggregates for six months and a bug is found in the calculation. Backfill six months while the live job keeps running, with no double counting and no gap. Describe the procedure and the exact cutover step.

Approach: Make each output a deterministic function of its inputs rather than of the order of writes, then separate the backfill writer from the live writer by version rather than by time.

Write the corrected six months into a separate versioned table, run both versions in parallel until they agree on a live overlap window, then repoint the readers in one atomic swap. There are four parts. First make each output partition an idempotent overwrite of a deterministic function of its inputs, so recomputing hour H replaces hour H; an aggregate written by appending or incrementing cannot be backfilled safely at all and has to be rebuilt into that shape first. Second run the backfill into pnl_v2 with the corrected code from the start of history up to a fixed watermark boundary set a few hours behind the live edge, so the two writers never contend for the same partition. Third deploy the corrected code to the live job writing pnl_v2 from that same boundary forward, and let it run a full day while a reconciliation compares v1 and v2 on the overlap and reports only the expected differences. Fourth swap the view that consumers read, which is the sole visible step and is a metadata operation. Keep v1 for a retention period so the swap can be reversed. This avoids the two real failures: a backfill writing the partition the live job is currently writing, and a consumer reading a half rewritten history and reporting a number nobody can reproduce.

Follow-up: How do you choose the retention period for the old version when consumers cache results downstream?

Key concepts: idempotent overwrite, partition versioning, atomic swap, watermark boundary.