Failover and Redundancy on a Trading Desk

Failover on a trading desk has to reconcile open orders and positions correctly rather than merely bring a backup process online, or the switch itself becomes the incident.

Failover on a trading desk means moving a strategy's live state, its open orders, its position, and its view of the book, from a failed primary process to a standby without either duplicating orders or losing track of a position, and the hard part is never bringing the standby online, it is proving the standby's state matches reality before it starts sending. A failover that trades on a wrong assumption about current position is often worse than the outage it was meant to prevent.

An active-passive design runs one live instance of a strategy and keeps a standby synchronised but idle, promoting it only when the primary is confirmed down. It is simpler to reason about because only one instance can ever send an order at a given moment, but it depends entirely on how quickly and how correctly the standby can be brought up to date.

An active-active design runs two instances simultaneously, each capable of trading, and relies on careful partitioning, by instrument or by order id range, to stop them from acting on the same decision twice. It removes the promotion delay entirely but raises the stakes on that partitioning logic, since a bug there produces duplicate orders rather than a clean gap in coverage.

Before a standby starts sending orders, it needs an accurate picture of every order currently working at the exchange, the firm's confirmed position, and the current state of the book. Open orders are recovered by querying the exchange directly for the account's working orders rather than trusting the primary's last known state, because the primary may have failed after sending an order but before recording that it sent it.

Position is reconciled against the exchange's own record, not against the firm's internal ledger alone, since the two can disagree exactly in the failure scenarios that make failover necessary in the first place. A standby that trusts internal state over the exchange's confirmed state can size a new order against a position that no longer exists.