Message Queues and Market Data Distribution

Market data reaches a firm over multicast and is redistributed internally over a message bus, and the slow consumer problem decides how that bus is designed.

Market data reaches a firm from the exchange as multicast UDP, and once it arrives, the firm has to redistribute the normalized version of it to every strategy, risk process and logging system that needs it, which is a distribution problem inside the firm's own network as well as at the exchange boundary. How that internal message bus handles a slow reader determines whether one underperforming process can degrade every other process sharing the bus.

Exchange multicast is built so that adding a new subscriber does not add load to the sender, since every subscriber on the network segment receives the same packets. Firms mirror that property internally: a feed handler publishes the normalized book once, and any number of strategies, risk services, and recorders subscribe to the same internal multicast group or shared memory segment without the feed handler needing to know how many consumers exist.

The design goal is the same at both layers, that the publisher's cost stays constant regardless of subscriber count, but the internal bus adds a requirement multicast at the exchange does not have to solve on the firm's behalf: guaranteeing that every internal consumer actually gets every message, since a firm cannot ask the exchange to retransmit data that already left its own internal network.

A slow consumer is a process that reads messages off the bus more slowly than they arrive, and the central design question in any message distribution system is what happens when that gap grows. One approach lets the slow consumer's queue grow until it runs out of memory and the process crashes, isolating the damage to that one consumer. A second approach applies backpressure that slows the publisher itself, which protects the slow consumer but can throttle every other, healthy consumer sharing the same feed.

Most production trading systems choose isolation over backpressure for market data specifically, accepting that a slow consumer may drop messages or fall behind and have to resynchronise, rather than letting one underperforming process add latency to a strategy that is otherwise keeping up. The same choice would be unacceptable for order flow, where every message matters and dropping one is not an option, which is why order and market data traffic are usually kept on physically separate paths.