A streaming job consumes 12 partitions and emits 5 minute windows on an event time watermark taken as the minimum across partitions. One partition carries a symbol that trades twice an hour. Explain what happens to the output of the whole job, and give two fixes with their costs.

A streaming job consumes 12 partitions and emits 5 minute windows on an event time watermark taken as the minimum across partitions. One partition carries a symbol that trades twice an hour. Explain what happens to the output of the whole job, and give two fixes with their costs.

Approach: Trace how the global watermark is derived from the per partition watermarks, then ask what the slowest partition does to a minimum and what would let the others advance honestly.

The whole job stalls: the global watermark is the minimum over partitions, so the quiet partition holds it back and no window from any partition is emitted until that symbol trades again, up to half an hour later. Taking the minimum is correct in principle, because a window can only close once every source has passed its end and an idle source has made no such claim. Fix one is an idleness timeout, where a partition producing nothing for a configured period is excluded from the minimum so the other eleven advance. The cost is that a tick arriving on that partition after it was marked idle is late by construction and lands in the late path, so the timeout must exceed the genuine quiet period, and window emission latency for the busy symbols is being bought with correctness on the quiet one. Fix two is a heartbeat from the source carrying its current time when it has no data, which advances that partition's watermark honestly at the price of one small message per interval and a change on the producer side; this is the better fix when you control the producer. The option to avoid is a processing time fallback for the job as a whole, which makes window contents depend on when the job ran and destroys replay reproducibility.

Follow-up: How do you derive the idleness timeout from the data instead of setting it by hand?

Key concepts: per partition watermark, idle source, window emission, processing time fallback.