Logging and Observability in Production Trading

Every order in a production trading system needs a reconstructable timeline from decision to fill, and that requirement shapes how logging and metrics are built.

Observability in a trading system exists to answer one recurring question under time pressure: what exactly happened to a given order, in what order, and on which host, between the moment a strategy decided to send it and the moment it was acknowledged or filled. That requirement, a fully reconstructable timeline per order, shapes logging, metrics and tracing differently than it would in a typical web application, because the cost of an unanswered question here is measured in dollars per second, not in a delayed bug report.

Every order carries a unique identifier the moment the strategy generates it, and that identifier is attached to every log line, metric and trace span the order touches across every host it passes through, the strategy process, the risk layer, the gateway, and back through the acknowledgement path. Without a consistent identifier threaded through all of that, reconstructing an order's path means correlating timestamps across systems by hand, which is slow and, given clock drift between hosts, often unreliable.

Logs capture the decision itself, the specific risk check that passed or failed and why, and the raw message sent to and received from the exchange, since a dispute over what the exchange actually received is resolved by comparing the wire message logged at send time against the venue's own record, not by memory or inference.

Metrics are numeric aggregates, order rate, fill rate, average and tail latency, sampled continuously and cheap enough to keep at very high resolution, which makes them the right tool for real-time dashboards and automated alerting. Logs are discrete, detailed records of individual events, expensive to store at full fidelity over long periods but essential for reconstructing exactly what one specific order did. Traces connect the two, following a single order or a single market data update across every process it touched, which is what makes it possible to see that a hundred-microsecond delay originated in the risk layer rather than the gateway.

Conflating these three, for example trying to debug a single bad fill using only aggregate latency metrics, wastes time chasing a system-wide average when the actual cause was one host with a stuck thread for eleven milliseconds.