Two threads each increment their own atomic counter. The counters are adjacent members of one struct. Total throughput is 20 million increments a second. Giving each counter its own 64 byte slot raises it to 300 million. Explain the mechanism and the one line fix.

Two threads each increment their own atomic counter. The counters are adjacent members of one struct. Total throughput is 20 million increments a second. Giving each counter its own 64 byte slot raises it to 300 million. Explain the mechanism and the one line fix.

Approach: Think about the unit the coherence protocol works in, and what happens to a single unit that two cores both want to write.

The two counters share one 64 byte cache line, so each increment on one core invalidates the line in the other core cache and the line moves back and forth between them, which is false sharing, and the fix is alignas(64) on each counter or 64 bytes of padding between them. Cache coherence works per line, so a write must take the line in the modified state and steal it from the other core, costing 30 to 100 nanoseconds each time against a few nanoseconds for an uncontended increment. The counters were never shared in the program logic and the hardware cannot see that, since it tracks addresses at line granularity. A 15 times gap is typical for this pattern, and perf c2c or the HITM event identifies the offending line by address.

Follow-up: Where does the same effect appear in a lock-free queue that keeps its head and tail indices in one struct?

Key concepts: false sharing, cache line, cache coherence, alignas.