7.1.5 Failure Modes & Real-World Practice

Basic Prerequisites: 04 · Non-Stationarity & Samples.

---

1. Intuition & Practical Objective

The pillar’s entire message compresses into one discipline: most financial-ML “discoveries” are false positives. This page names the failure modes precisely, ties each to a first principle, and - where possible - measures how big the fraud is. The objective is not cynicism; it is the muscle of knowing which numbers to trust.

The failure modes, in one line each:

1. Overfitting with few effective samples - flexible model + tiny $N_{\text{eff}}$ = memorizing noise (01, 03).
2. Random / leaky cross-validation - IID CV on non-IID, overlapping-label data inflates the score (02).
3. Look-ahead (point-in-time) bias - using information that was not available at the timestamp (reported earnings used at the open; today’s constituents backtested into the past).
4. Survivorship bias - backtesting only today’s S&P 500 constituents excludes the Enrons and Lehmans that died along the way.
5. Selection bias / multiple testing - the best of $N$ tried strategies looks great by pure chance (expected max Sharpe $\approx \sqrt{2\ln N}$).

The one-line takeaway. “Ask yourself, in what way am I overfitting? - overfitting is not a tuning nuisance, it is the industry’s default mode of operation and, when done knowingly, outright fraud (López de Prado).“

---

2. Mathematical Ground Truth & Derivations

Wrong-vs-right CV (ESL §7.10.2). If features are screened on the full dataset (e.g., pick the most correlated features using all rows, then cross-validate), the test fold’s information has already leaked into the feature choice. ESL’s canonical demo - 50 samples, 1000 irrelevant predictors, true $R^2=0$ - shows the leaky procedure reporting an average CV error of $3\%$ while the honest (screen-inside-each-fold) procedure reports the true $\sim 50\%$. Screening must happen inside each training fold only.

Selection bias / multiple testing (Bailey & López de Prado 2014). If you try $N$ skill-less strategies and keep the best, its in-sample Sharpe ratio is not $0$; it is (asymptotically, for IID Normal returns) the expected maximum of $N$ standard Normals,

$$\mathbb{E}\left[\underset{i=\mathrm{1..}N}{\max }{\mathrm{SR}}_i\right]\approx \sqrt{2\ln N},$$

in annualized units scaled by the track length. To prevent skill-less strategies from showing an in-sample Sharpe of ${\mathrm{SR}}_{\text{IS}}$, you need a Minimum Backtest Length

$$y\ge \frac{2\ln N}{{\mathrm{SR}}_{\text{IS}}^2}\text{years}.$$

The simple bound $y\ge 2\ln N/{\mathrm{SR}}_{\text{IS}}^2$ is conservative: for $N=45$, ${\mathrm{SR}}_{\text{IS}}=1$ it demands $\approx 7.6$ years. The exact Bailey Minimum-Backtest-Length $y=(E[{\max }_{N}Z]{)}^2/{\mathrm{SR}}_{\text{IS}}^2$ with $E[{\max }_{N}Z]=(1-\gamma ){\Phi }^{-1}(1-1/N)+\gamma {\Phi }^{-1}(1-1/(Ne))$ is sharper - $N=45$ gives $E[\max ]=2.235\Rightarrow y=5.0$ years (and $N=7\Rightarrow 1.92$ yr, matching the paper’s two-year example). So the same bookkeeping says: over a 5-year backtest, no more than $\approx 45$ independent configurations should be tried (the corpus-verified Bailey figure: “if only five years of data are available, no more than forty-five independent model configurations should be tried”).

---

3. Computational Implementation - two failure modes, measured

Experiment A - wrong-vs-right CV (ESL 7.10.2). On 50 samples × 1000 irrelevant features (true $R^2=0$), screening on the full data vs screening inside each fold.

Experiment B - selection bias. Try $N=200$ pure-noise strategies (each one year of random returns), keep the best, and see its in-sample Sharpe.

Experiment A reproduces the ESL wrong-vs-right lesson in $R^2$ units: leaky screening reports a spurious $+0.047$ (looks predictive on pure noise) while honest screening shows the model is worthless ($-0.89$). Experiment B is the selection-bias killer: of 200 pure-noise strategies, the best shows an in-sample annualized Sharpe of 2.73 - and 40 of the 200 look “good” (SR>1) by pure chance. If you tried 200 strategies and are pitching the best one, its Sharpe is most likely this artifact, not skill.

---

4. Failure Modes & First-Principles Breakdowns (numbered)

1. Leaky cross-validation / screening. Screening features or scaling on the full dataset before splitting leaks the test fold (ESL §7.10.2; AFML Ch. 7). Fix: preprocessing and feature selection inside each training fold only.
2. Look-ahead (point-in-time) bias. Using data at a timestamp before it was actually available (a 10-Q filed at 16:30 used at 09:30; today’s index constituents backtested to 2005). The fix is strict point-in-time (PIT) datasets - see Alt-Data Pipelines & Evaluation.
3. Survivorship bias. A universe filtered to today’s S&P 500 drops Enron, Lehman, WorldCom - the very names that reveal a model’s flaws. Backtest on point-in-time constituent lists.
4. Selection bias / multiple testing. Tuning over $N$ configurations inflates the best in-sample Sharpe to $\approx \sqrt{2\ln N}$ (Experiment B). Report $N$ and use the Deflated Sharpe (06).
5. The “Sisyphus paradigm” (AFML Ch. 1). Demanding every quant independently produce a strategy in six months guarantees each settles for an overfit backtest or crowded factor investing. The cure is the team-based meta-strategy paradigm, not more solo searching.

---

5. References

• López de Prado, Advances in Financial Machine Learning
• Bailey, David H. & López de Prado, Marcos, “Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance,” Notices of the AMS 61(5), 2014
• Hastie, Tibshirani & Friedman, The Elements of Statistical Learning

---

6. Connected Graph Bridges

• Back: 04 · Non-Stationarity & Samples · Index Hub
• Forward: 06 · Advanced Extensions
• Sibling solutions: Backtesting Hygiene & Deflated Sharpe · Purged & Embargoed CV · Alt-Data Pipelines (point-in-time hygiene)