A leader holding a 10 second lease is paused by a 20 second garbage collection. The lease expires, a new leader is elected and starts writing, and the old leader then wakes and writes too. Explain why a lease alone cannot prevent the corruption and give the mechanism that does.

A leader holding a 10 second lease is paused by a 20 second garbage collection. The lease expires, a new leader is elected and starts writing, and the old leader then wakes and writes too. Explain why a lease alone cannot prevent the corruption and give the mechanism that does.

Approach: Ask what the paused process believes when it resumes and whether any check it can perform locally would tell it the truth, then move the check onto the resource being written.

A lease is checked by its holder against its own clock, and a paused process resumes believing the lease is still valid, so the only reliable defence is a fencing token: a monotonic epoch number issued with the lease and checked by the storage resource, which rejects any write carrying a token below the highest it has seen. The pause is the whole problem. Between the lease expiry and the moment the old leader resumes it has no way to learn that time passed, and any local check it performs happens before the write it has already decided to send. This is the split brain scenario, and both writers behave correctly by their own information. With a fencing token the new leader holds epoch 8 and the old holds epoch 7; the resource records 8 on the first new write and refuses the stale write from 7. The requirement this places on the design is that the resource must participate, so a plain object store or a file system with no conditional write cannot be fenced and needs a compare and set placed in front of it. Shortening the lease narrows the window and never closes it, because a pause can always exceed any finite lease.

Follow-up: What do you do when the resource supports a conditional write on one object but the leader must update ten of them together?

Key concepts: fencing token, lease expiry, split brain, monotonic epoch number.