Where Machine Learning Actually Works in Trading

Machine learning helps most in execution, order sizing, and feature selection, and least at predicting next day equity direction from price alone.

Machine learning earns its place in trading in a narrow set of jobs: execution, order sizing, feature selection across wide alternative data sets, and picking up non-linear interactions in slower signals that decay over days rather than seconds. It does not reliably find a new directional edge in daily equity returns on its own, because the signal in that data is small relative to the noise, and a general purpose learner given raw prices and told to predict tomorrow's return will mostly fit noise.

A researcher arriving from image recognition or natural language processing is used to models that explain 80 percent or more of the variance in a well posed problem. Daily equity return prediction rarely clears an R squared of 1 percent out of sample, and a figure above 5 percent on a large universe should be treated as a bug in the backtest before it is treated as an edge. This is not a statement about the quality of any particular algorithm. It is a statement about the data generating process: a market absorbs public information quickly, and what is left over for a model to explain is a thin residual sitting inside a large amount of noise.

The practical consequence is that model selection has to be biased toward low variance estimators. A method that reduces bias at the cost of variance, the way a large neural network does on a small, noisy tabular data set, will overfit before it improves accuracy. A method that trades a little bias for a large reduction in variance, the way a shallow gradient boosted tree with heavy regularization does, tends to hold up better once real money is on the line.

Execution is the clearest case. Predicting the next few minutes of order flow, queue position, or the probability that a resting order fills before the market moves away from it is a much easier statistical problem than predicting tomorrow's return, because the relevant history is dense, the label arrives quickly, and the mechanism, other participants reacting to visible order book state, is genuinely present in the data. Firms including Citadel Securities, Jump Trading, and Hudson River Trading built market making franchises on exactly this kind of short horizon, high frequency modeling, described in more detail in our guide to high frequency trading and market microstructure.

Order sizing is the second case. Once a signal exists from any source, deciding how much capital to commit given the current estimate of edge, volatility, and correlation with the rest of a book is a well defined optimization problem with a large, clean sample of realized outcomes. A tree based model conditioning bet size on recent volatility regime, spread, and inventory tends to beat a fixed fractional rule, and the improvement is measurable because the objective, risk adjusted return, is stated plainly rather than inferred.