Reinforcement Learning for Execution and Its Constraints

Reinforcement learning suits optimal execution because the problem is sequential with fast feedback, but reward design and simulator gaps limit it.

Reinforcement learning for execution means training an agent to break a large order into smaller pieces and time their placement to minimize cost against a benchmark such as arrival price or the volume weighted average price, and it earns a real place in this narrow problem because the agent gets thousands of decisions and fast, clear feedback within a single trading day. Its constraints are just as concrete: reward design is difficult to get right, and the gap between a historical simulation and live trading is large enough that most of the engineering effort goes into closing it rather than into the learning algorithm itself.

Optimal execution is naturally sequential: an agent holding a remaining quantity to work observes the current book state, chooses how much to send and at what price, watches the outcome, and repeats, potentially hundreds of times before an order is fully worked. This structure, a sequence of decisions with a delayed cumulative reward, is exactly what reinforcement learning is built for, in contrast to the single shot prediction problems that dominate the rest of this discussion.

Classical execution models, most notably the framework built around trading off market impact against timing risk, give a closed form schedule under simplifying assumptions about how the market responds to trading. A reinforcement learning agent can, in principle, learn a schedule that adapts to the current book depth, spread, and short term volatility in ways the closed form solution does not, which is the appeal for a desk running significant execution volume.

An agent trained to minimize implementation shortfall relative to the arrival price will learn to trade slowly when that reduces cost and aggressively when urgency is rewarded, but only if the reward function actually reflects the tradeoff the desk cares about. A reward that only penalizes slippage without any penalty for unfinished inventory at the end of the trading window will learn to sit and wait indefinitely. A reward that heavily penalizes variance in outcomes can produce an agent that trades too passively and misses favorable prices.

Getting this specification right takes several iterations, each one requiring a full retraining run and a careful look at the resulting policy's behavior across a range of market conditions, in addition to its average reward. A policy with a good average reward and a rare but severe failure mode, dumping a large remaining quantity into a thin book near the close, is not acceptable on a live desk even if the average number looks good.