One malformed message makes a consumer throw, and it retries forever, so that partition stops advancing while the other 11 stay healthy. Describe the dead letter design that unblocks it, and state exactly what must be recorded for the message to be replayable later.

One malformed message makes a consumer throw, and it retries forever, so that partition stops advancing while the other 11 stay healthy. Describe the dead letter design that unblocks it, and state exactly what must be recorded for the message to be replayable later.

Approach: Bound the retry, move the failing message somewhere durable before advancing past it, and ask what a replay months later would need in order to reconstruct the same decode.

Retry a bounded number of times with backoff, then write the raw message with its context to a dead letter queue, advance the offset past it, and alert on the dead letter rate. The retry has to be bounded because the failure is deterministic: a malformed payload fails identically on every attempt, so an unbounded retry converts one bad message into an outage on that partition, which is why the symptom is growing lag on exactly one partition out of twelve. The offset advance is what unblocks it and is safe only once the message is durably stored elsewhere. For a replay to be possible, record the raw bytes exactly as received, the topic, partition and offset, the schema id in force, the consumer version, the receive timestamp, and the exception with its stack. Without the raw bytes a replay reprocesses a parsed approximation of the message, and without the schema id the same bytes may decode differently after a registry change. Separate two causes when alerting: a genuinely malformed message, and a transient dependency failure that a short retry budget misclassifies as poison, which is why the budget should be long enough to cover a database restart.

Follow-up: How do you replay a dead letter safely when downstream state has already moved on by a day?

Key concepts: dead letter queue, bounded retry, offset advance, replay metadata.