A colleague reports 62% out-of-sample accuracy from 10-fold random cross-validation. The features include 20-day rolling means and the label is the sign of the next 5-day return. Name every way the estimate is inflated and give the split design that fixes it.

A colleague reports 62% out-of-sample accuracy from 10-fold random cross-validation. The features include 20-day rolling means and the label is the sign of the next 5-day return. Name every way the estimate is inflated and give the split design that fixes it.

Approach: Track which calendar days enter each feature and each label, then find every pair of training and test rows that share days or violate the direction of time.

Random folds allow leakage on both the label horizon and the feature window, so the estimate is contaminated and must be replaced by a forward-chaining split with purging and an embargo. The label spans 5 days, so a training row can share up to 4 of its label days with a test row, and the features are 20-day rolling means, so training rows can be built from data dated after the test row. Random folds also let the model use the future to predict the past, which no live deployment can do, and any scaling, winsorising or feature selection carried out before the split spreads test information across the whole sample. The fix is a forward-chaining split: train on days 1 to t, test on days after t, roll forward, purge from training every observation whose label window overlaps the test window, and embargo at least 20 further days to cover the feature window. Expect the corrected figure to fall well below 62%, which on a sign label is close to the base rate of up days anyway.

Follow-up: With 8 years of daily data and a model that needs 3 years to fit, how many independent evaluation folds does a forward-chaining design actually give you?

Key concepts: data leakage, purged cross-validation, embargo, forward chaining.