Every quant paper claims out-of-sample results; few earn the phrase. “Out-of-sample” is not a data split — it’s a protocol, and the protocol’s entire value comes from what you didn’t do: didn’t look, didn’t iterate, didn’t quietly try again. This guide covers the mechanics (splits, purging, walk-forward structure) and the discipline, which is harder than the mechanics.

Why the ML playbook fails on market data

Standard k-fold cross-validation assumes exchangeable observations. Financial time series violate this three ways, each one a leak:

  1. Serial dependence: adjacent observations share information — training on Tuesday and testing on Wednesday is nearly testing in-sample, especially with overlapping feature windows or multi-day labels.
  2. Boundary leakage: a label looking k periods ahead, computed near a split boundary, contains test-period outcomes. Train/test contamination via labels is the look-ahead family’s sneakiest member — we flagged exactly this (100-event labels, 9-tick window overlap at an 80/20 boundary) in a published replication review.
  3. Random shuffling destroys regimes: shuffled folds train on 2022 to predict 2019, giving every fold knowledge of all regimes — an option live trading never has.

The fixes, standard since López de Prado formalized them: purging (drop training samples whose labels overlap the test window) and an embargo (exclude a further buffer — at least feature-window + label-horizon — after each test block). If you use CV on market data at all, it’s purged, embargoed, and time-ordered, or it’s leaking.

The walk-forward structure

Walk-forward is the honest default because it mimics deployment: fit on the past, trade the next slice, roll forward, never look back.

anchored:   [train ........][test]
            [train ..............][test]
rolling:        [train ....][test]
                    [train ....][test]
  • Anchored (expanding window) uses all history — more data, slower adaptation, implicitly assumes old regimes stay relevant.
  • Rolling (fixed window) adapts and forgets — the window length becomes a hyperparameter with regime-size meaning, and results that hinge on it deserve the same plateau-vs-spike scrutiny as any other parameter.
  • Either way, re-optimize inside each training window only — parameters chosen once on the full sample then “walk-forward tested” is in-sample fitting wearing a lab coat.

What to read from the output: not just aggregate performance but stability across windows (one golden window carrying the average is a regime bet, not an edge — check the subperiod table), and parameter drift across refits (parameters that lurch every window are chasing noise). The ML and RL literatures fail these two checks more than any others.

The evaluate-once rule

Here is the uncomfortable truth that makes protocols matter: a holdout is spent the moment it influences a decision. Evaluate on the holdout, tweak, evaluate again — after a few cycles the holdout is training data with ceremony, and its Sharpe is quietly the maximum of several tries. The discipline:

  • Decide the final test protocol and success criteria in writing before touching the holdout (Gate 1 requires exactly this).
  • Spend it once. If the result fails and you keep iterating, carve a new, never-touched period for the next final exam — and count the spent one in your trial count.
  • Log every evaluation. An honest lab notebook of holdout touches is the cheapest integrity infrastructure that exists.

The out-of-sample tests nobody can fake

Two sources of OOS data are immune to your own iteration. Post-publication data for published strategies: a 2015 paper meets 2016–2026 markets it has never seen — the cleanest test in the field and the heart of reproduction work. And the future: paper trading through your real execution path, then minimum-size live trading, per the go-live protocol. Every backtest, however disciplined, shares your assumptions about fills and costs; the fill-quality gap between simulation and live is itself the final out-of-sample statistic.

The checklist

  • ☐ Time-ordered splits only; no shuffling
  • ☐ Purge overlapping labels; embargo ≥ window + horizon at every boundary
  • ☐ All transforms fit inside training windows (scalers included)
  • ☐ Re-optimization inside each walk-forward window
  • ☐ Per-window results and parameter paths reported, not just the aggregate
  • ☐ Holdout criteria written down first; holdout spent once; touches logged
  • ☐ Trial count carried into significance judgment

The multiple-testing arithmetic: Deflated Sharpe ratio → · Where leakage hides: look-ahead taxonomy → · From the corpus: XGBoost in finance: what the papers show →