Neural Networks in Quant Research and Their Practical Limits

Neural networks earn their place in high frequency order book data and unstructured alternative data, and struggle on small, noisy daily tables.

Neural networks in quant research earn a place wherever the raw data has structure a tree based model cannot exploit directly, images, text, audio, or long high frequency sequences, and wherever the available sample size runs into the millions of observations rather than the thousands. Their practical limit is the opposite case, daily tabular market data with a modest number of rows, where their extra flexibility mostly buys overfitting rather than accuracy.

Tick by tick order book data for a liquid instrument produces millions of events in a single trading day, and a firm running high frequency strategies at Jump Trading, Hudson River Trading, or Tower Research Capital has access to a sample size that changes which model families are worth considering. At that scale a neural network's usual weakness, needing far more data than a tree ensemble to reach a stable fit, stops being the binding constraint, and its ability to model long sequences and complex temporal dependencies becomes the more relevant property.

This is a genuinely different regime from daily equity return prediction, and conflating the two, applying lessons learned from high frequency microstructure modeling to a slower daily signal with a few thousand rows, is a common source of disappointment when a technique that worked at one desk fails to transfer to another problem at the same firm.

Recurrent architectures and their more recent attention based successors are a natural fit for predicting short horizon order book dynamics, queue depletion, and the probability that a resting order fills, because the input is genuinely sequential and the relevant history extends across many time steps in a way a tree ensemble, which treats each engineered feature independently unless the researcher hand builds lag features, does not capture as naturally.

The label in this setting, whether an order fills in the next few seconds or whether the mid price moves by a tick, arrives quickly and in enormous volume, which supplies exactly the sample size these models need. This is the microstructure territory covered in limit order book trading mechanics, and it is where a firm's investment in a sequence model is most likely to be repaid.