A trading system runs a primary and a hot backup. The primary stops sending heartbeats. Describe the failover decision, the two ways it goes wrong, and how sequence numbers and cancel-on-disconnect bound the damage.

A trading system runs a primary and a hot backup. The primary stops sending heartbeats. Describe the failover decision, the two ways it goes wrong, and how sequence numbers and cancel-on-disconnect bound the damage.

Approach: Consider that a missing heartbeat is ambiguous between a dead primary and a slow network, then design for both readings being wrong.

The backup must not take over on a missed heartbeat alone, because the failure is indistinguishable from a network partition, and taking over blindly produces split brain with two systems quoting the same book. The first failure mode is exactly that: the primary is alive and unreachable, the backup activates, and both send orders against one position, so the firm doubles its intended size with neither side aware. The second is the opposite, a backup that is too conservative and never activates, leaving live orders in the market with nothing managing them. The resolution is that the exchange session, rather than either host, arbitrates the failover. Each session carries a sequence number, so only one connection can be the sender for that session and a second login either displaces the first or is rejected, which makes the exchange break the tie. Cancel-on-disconnect instructs the exchange to pull the primary's resting orders the moment its session drops, so the market is flat before the backup starts. The backup then logs in, reads the sequence number and the drop copy to rebuild the true position, and only quotes once its state matches the exchange's. State replication between the two hosts should be a log the backup replays, so recovery is deterministic rather than a guess.

Follow-up: Cancel-on-disconnect fires and the primary reconnects 40 milliseconds later believing its orders are live. What in your design prevents it from acting on that belief?

Key concepts: split brain, failover, heartbeat, cancel on disconnect.