7.2 Purged Cross-Validation & Backtest Hygiene

Basic Prerequisites: Probability & Measure Theory and Backtesting Hygiene (the Deflated Sharpe Ratio, PBO (probability of backtest overfitting), multiple testing). Comfort with Feature Engineering & Labeling (interval labels) is what makes the purge meaningful. (these are the folder-level prerequisites for pages 02–06; page 01 states its own, smaller, entry requirements)

1. Intuition & Practical Objective

Every ML evaluation in finance is an act of leakage control. A standard $k$-fold cross-validation (CV) splits an IID dataset into train and test and shuffles so that “no observation leaks into another set.” But financial observations are not IID: features are serially correlated ($X_t\approx X_{t+1}$) and, because labels are formed over overlapping forward windows (e.g., “sign of the next $h$-bar return”), adjacent labels are near-duplicates ($Y_t\approx Y_{t+1}$). Put $t$ in the training set and $t+1$ in the test set and you have just told the model the answer.

This folder is the topic-hub for purged cross-validation and backtest hygiene in Kwant-Atlas, the companion to the classical backtesting-hygiene folder (multiple testing, DSR) but for the ML pipeline. It (a) gives the fast formula/decision lookup below - job #1 of a hub - and (b) routes you to six sub-pages that go from raw intuition through the two leak-removal tools (purging, embargoing), their combinatorial generalization (CPCV), and the failure modes and practice.

The one-sentence essence. “Standard $k$-fold CV measures skill plus leaked duplicate labels; purging drops every training observation whose label window overlaps the test set, an embargo drops the serially-correlated stragglers right after it, and Combinatorial Purged CV (CPCV) repeats the honest split across every test-set combination to yield a distribution of backtest paths instead of one fragile number.”

2. Mathematical Ground Truth & Derivations

Quick-Reference Lookup (job #1). Notation: label of observation $i$ spans the closed bar interval $[t_{i,0},t_{i,1}]$; test block spans $[t_{j,0},t_{j,1}]$; $T$ = number of observations, $N$ = number of contiguous groups, $k$ = number of groups in the test set, $h$ = embargo length in bars, $\phi$ = number of backtest paths. All checks below were re-executed and reproduced exactly (see §3).

Critical caveat (AFML §12.4.3). There is a strict budget: more paths $\phi$ requires either more groups $N$ (smaller group granularity) or a bigger test fraction $k/N$ - and the latter shrinks the training data ($\theta =1-k/N$). The practical sweet spot is $k=2$ with $N=\phi +1$, which trains on $\theta =1-2/N$ of the data while still yielding $N-1$ paths. Pushing to $k=N/2$ maximizes $\phi$ but trains each classifier on only half the sample.

3. Computational Implementation - the hygiene engine

Standard-library only. This reproduces the two headline numbers in §2: the leakage signature (accuracy inflates with the number of folds on a pure-noise feature) and the CPCV path counter.

Read it as the punchline of the whole folder: on pure noise, thinning the folds from block-$\sim 478$ to leave-one-out drives standard CV accuracy from $0.45$ to $0.85$ - a textbook “false discovery” manufactured purely by leaked overlapping labels. Purging pins every configuration at $\approx 0.5$. The same classifier sees no signal.

4. Failure Modes & First-Principles Breakdowns

Hub signposts - the practice checklist lives in 05 · Failure Modes & Practice. In one line each:

1. Look-ahead (label) leakage - overlapping label windows make adjacent train/test samples near-duplicates; standard CV reads the answer back ($\S$02).
2. Purging is not free - dropping overlapping training rows shrinks the sample, and the loss grows with the label horizon (up to $11\%$ at $h=50$); a real bias–variance tradeoff ($\S$05).
3. One path = one lottery ticket - walk-forward and CV each produce a single backtest, so their Sharpe is a high-variance draw; CPCV replaces it with a distribution ($\S$04).
4. The missing $N$ again - a purged CV that is run a thousand times still suffers selection bias; pair it with the Deflated Sharpe Ratio (Backtesting Hygiene).

5. References

• López de Prado, Marcos: Advances in Financial Machine Learning (Wiley, 2018)
• Bailey, Borwein, López de Prado & Zhu: The Probability of Backtest Overfitting, J. Computational Finance 20(4) (2017)
• Bailey & López de Prado: The Deflated Sharpe Ratio, J. Portfolio Management 40(5) (2014)
• Hastie, Tibshirani & Friedman: The Elements of Statistical Learning (2nd ed., 2009)
• “Backtest Overfitting in the Machine Learning Era” (Expert Systems with Applications, 2025)

6. Connected Graph Bridges

• Foundational base: Probability & Measure Theory · Backtesting Hygiene (classical: DSR, PBO, multiple testing)
• Sibling (same pillar): Tree-Based Factor Ranking & Purged CV (purging applied to LightGBM/XGBoost MDA) · Financial ML Pitfalls & Low SNR
• Labels that need purging: Feature Engineering & Labeling (triple-barrier & interval labels)
• Sub-pages (in-folder): 01 From Zero · 02 Why Standard CV Fails · 03 Purging & Embargo · 04 Combinatorial Purged CV · 05 Failure Modes · 06 Advanced Extensions

Beginner: start at 01 · Practitioner: start at 05