Deployment Practices for Trading Software

A deploy that reaches only part of a trading fleet is a known failure mode, and the 2012 Knight Capital loss remains the clearest documented example of it.

Deploying new code to a trading system carries risk that an equivalent deploy to most other software does not, because the code being deployed is directly authorised to move money the moment it starts running, with no human review step between a decision and an order reaching the exchange. Staged rollouts, canary hosts, and a strict rule against leaving old code paths reachable are the standard practices that keep a deploy from becoming the incident itself, and the 2012 failure at Knight Capital remains the clearest documented case of what happens when those practices are skipped.

A deploy changes the behaviour of a system that is, by design, allowed to act without waiting for confirmation, so a bug introduced by a deploy does not wait to be discovered by a user complaint, it starts sending orders on the first tick after the new code goes live. That makes the deployment process itself a control surface as important as any single line of strategy code, and firms treat it accordingly, with sign-off procedures, staged exposure, and an explicit rollback plan required before any change reaches production.

A canary rollout pushes new code to a single host first, often trading the smallest size or the least liquid instrument in the strategy's universe, and holds it there under close observation before promoting the same code to the rest of the fleet. The canary period is short by design, minutes rather than hours, but it is long enough to catch a class of bug that unit tests do not: a wrong assumption about live market conditions that only shows up once the code is actually trading.

Staged rollouts extend the same idea across the full fleet, moving from the canary host to a small percentage of production, then to a larger percentage, with monitoring checked at each stage before proceeding, rather than pushing to every host simultaneously.

A deploy that reaches part of the fleet but not all of it, whether by design during a staged rollout or by accident when a deployment script fails partway through, leaves the firm running two different versions of the same strategy simultaneously, often without every engineer on the desk realising it. If the two versions disagree about position, about a risk limit, or about which code path is currently active, the firm can end up in a state no single version was ever tested in, which is exactly the condition that turned a partial deploy into the largest documented deployment failure in the industry.