A service passes every test and corrupts one order in ten million in production, always during the busiest minute of the day. Say what class of bug this is, why the tests miss it, three ways to make it reproduce, and the tool that proves the cause.

A service passes every test and corrupts one order in ten million in production, always during the busiest minute of the day. Say what class of bug this is, why the tests miss it, three ways to make it reproduce, and the tool that proves the cause.

Approach: Ask what changes about thread interleavings once the machine is saturated, and which analyses detect an incorrect ordering rather than an incorrect output.

It is a data race whose window is a few instructions wide, so it needs two threads to interleave inside that window, which happens only once cores are contended and a thread is preempted at the wrong point; reproduce it by running under the thread sanitiser, by inserting delays or a scheduling fuzzer at the suspect points, and by running with fewer cores than threads to force preemption, and prove it with the thread sanitiser report naming the two conflicting accesses and the missing happens-before edge between them. The tests miss it because a lightly loaded run almost never lands a preemption inside the window, and passing a million times is weak evidence about a one in ten million event. The thread sanitiser flags the missing synchronisation on any execution that touches both accesses, so it does not need the bad interleaving to occur, at 5 to 15 times slowdown and 5 to 10 times memory. Drive it with a load generator at 3 times the production rate.

Follow-up: What would you add to the code so that the same class of bug fails a test deterministically in future?

Key concepts: data race, thread sanitiser, happens-before, preemption.