A model validated correctly on 2015 to 2022 data loses money in live trading from month one, with no coding error found. Give three distinct mechanisms that produce this, state a measurement that distinguishes them, and say what each implies about the fix.

A model validated correctly on 2015 to 2022 data loses money in live trading from month one, with no coding error found. Give three distinct mechanisms that produce this, state a measurement that distinguishes them, and say what each implies about the fix.

Approach: Separate the prediction from the execution. Check whether the forecast quality itself degraded, whether the costs of acting were understated, and whether the act of trading moved the price.

The three mechanisms are distribution shift in the features or the relationship, understated transaction costs, and market impact from the strategy's own trading, and they are distinguished by measuring the live prediction quality separately from the live profit and loss. Compute the information coefficient of the live forecasts against realised returns and compare it with the backtest value: if the correlation holds up while the profit does not, the forecast is fine and the loss is in execution, so the answer lies in costs or impact rather than in the model. Then compare realised fill prices with the decision prices used in the backtest to get the true cost per trade, and check whether the shortfall scales with traded size, which is the signature of impact rather than a fixed spread assumption. If instead the live information coefficient has collapsed, test whether the feature distributions have moved, which is covariate shift and is repairable by retraining on recent data, or whether the feature distributions are unchanged while the relationship to the target has changed, which is concept drift and means the edge itself has gone. Rule out an overfitted backtest as well, by comparing live results against the degradation expected from the number of configurations searched. Each diagnosis implies a different action: retrain, re-cost and resize, trade smaller, or stop.

Follow-up: How would you build a live monitor that flags covariate shift before the profit and loss reveals it, and what statistic would you compute per feature?

Key concepts: distribution shift, transaction costs, market impact, covariate shift.