1.2.2 Why Backtests Lie

Basic Prerequisites: 01 · From Zero (a backtest is a search) and Probability & Measure Theory (expectation, bias–variance).

---

1. Intuition & Practical Objective

Every good-looking backtest lies through one of three distinct doors, and hygiene means closing all three:

1. Selection bias - you report the winner of a search and hide the losers. (Multiple testing; the subject of 03 and 04.)
2. In-sample optimism - you measured performance on the data you tuned to. Even a single fitted model reports an optimistically biased error (Hastie et al., ESL Ch 7).
3. Contamination of the sample itself - look-ahead (using information not yet available), survivorship (today’s constituents), and backfill/self-selection (only successful track records published).

The practical objective: recognise which door a given number came through, so you know exactly how it overstates and by roughly how much.

The cleanest field evidence is the in-sample / out-of-sample gap. A rule optimised on 500 days of a pure random walk wins an in-sample Sharpe of $\approx 1.3$ and then delivers $\approx 0$ out of sample - because on a random walk there is no rule; only the search exists.

---

2. Mathematical Ground Truth & Derivations

2.1 In-sample optimism (ESL Ch 7)

For a fitted model, training error is systematically below test error. ESL quantify the gap for linear models:

• Bias–variance decomposition (ESL eq. 7.9): $\mathbb{E}[(Y-\hat{f}{)}^2]={\sigma }_{\epsilon }^2+{\mathrm{Bias}}^2+\mathrm{Var}$ - the irreducible ${\sigma }_{\epsilon }^2$ cannot be fitted away.
• In-sample variance of the fit (ESL eq. 7.12): $\mathrm{Var}$ in sample scales as $\frac{p}{N}{\sigma }_{\epsilon }^2$ with $p$ effective parameters and $N$ observations.
• Optimism of the training error (ESL §7.4, eq. 7.24): for a model with $d$ inputs/$d$ degrees of freedom,

$$\mathbb{E}[{\text{Err}}_{\text{in}}]\approx \mathbb{E}[{\text{Err}}_{\text{out}}]-\frac{2d}{N}{\sigma }_{\epsilon }^2,$$

• Effective degrees of freedom for a linear smoother $\hat{y}=Sy$ is $df=\mathrm{tr}(S)$ (ESL eq. 7.32) - this is the honest "$d$" to count, not the nominal parameter count.

Translation to finance: every free knob (lookback, threshold, stop, universe, weighting) raises $d$ and quietly lowers the reported in-sample error. A backtest Sharpe computed in sample is ${\text{Err}}_{\text{in}}$, i.e. it is biased optimistic by roughly $2d/N$ in variance units.

2.2 The IS → OOS gap and performance degradation

Let ${\widehat{SR}}^{\text{IS}}$ be the selected strategy’s in-sample Sharpe and ${\widehat{SR}}^{\text{OOS}}$ its Sharpe on unseen data. Define the performance degradation $\Delta ={\widehat{SR}}^{\text{OOS}}-{\widehat{SR}}^{\text{IS}}\le 0$ typically. The magnitude of $\Delta$ is driven by how much freedom the search had relative to the sample length: $\Delta \downarrow$ as the trial-space grows and $T$ shrinks.

Memory effects make it worse than “zero”. In an i.i.d. (memoryless) series a spurious pattern is merely diluted out of sample. In a series with memory (mean reversion, autocorrelation), a rule fitted to the extreme in-sample pattern must be undone when the process reverts - so the overfit rule does not earn zero, it earns negative returns systematically. Bailey, Borwein, López de Prado & Zhu (2014) prove that under memory effects backtest overfitting becomes loss maximization, not merely neutral. Most financial series are neither i.i.d. nor trendless, so the realistic sign of $\Delta$ is negative.

2.3 The three-door taxonomy (what each looks like numerically)

Door	Mechanism	In-sample symptom	Fix
Selection bias	Reported = max of $N$ trials	”Best” beats all others; $N$ undisclosed	DSR / haircut (pages 03–04)
In-sample optimism	Tuned on the scored data	$d$ free params raise apparent fit	Hold out + effective $df=\mathrm{tr}(S)$
Look-ahead	Uses future info	Unrealistically smooth equity curve	Point-in-time data, lag by publish date
Survivorship	Only survivors measured	Universe biased to winners	Point-in-time universe, include delistings
Backfill / self-selection	Only winners disclosed	Too-good-to-be-true track records	Demand full trial history

---

3. Computational Implementation - the IS/OOS collapse

A backtest on a pure random walk (zero drift, no memory, by construction no signal). We search a grid of 72 moving-average-crossover strategies (fast $\times$ slow $\times$ direction), pick the in-sample best, and score it out of sample. Stdlib only.

The searched rule looks like a Sharpe 1.3 strategy in sample and is worth exactly nothing out of sample. Nothing decayed and nothing broke: the data was a random walk, so the in-sample edge was the height of the maximum of 72 noisy draws (compare 03). This is the mechanism behind the “common practice of discounting reported Sharpe ratios by 50%” (Harvey & Liu 2015) - except the discount is non-linear, not a flat half.

---

4. Failure Modes & First-Principles Breakdowns

1. Tuning on the test set. Any parameter chosen using the test window converts it into a training window. Repeated holdout use makes 5% false positives expected after ~20 applications (Bailey & López de Prado §“holdout”).
2. Ex-post universe construction. “I backtested the S&P 500” using today’s members imports survivorship: firms that were delisted or acquired are missing, and membership required success.
3. Look-ahead leakage via stale vintages. Using restated fundamentals, index membership known only later, or a data vendor’s current adjusted prices, silently hands the strategy future information.
4. Ignoring effective $df$. Counting named parameters but not the implicit degrees of freedom of a search (a grid of 72 crossovers is 72 $d$‘s) understates optimism.
5. Assuming $\Delta \approx 0$. In mean-reverting markets the OOS Sharpe of an overfit rule is negative, so “past performance does not guarantee future results” is too lenient - adverse outcomes are likely (B&LdP §conclusion).
6. Confusing selection bias with model error. A low OOS Sharpe can mean (a) the strategy is a fluke, or (b) execution costs were mis-modelled. Hygiene analyses both; the fix differs.

---

5. References

• Hastie, Tibshirani & Friedman: The Elements of Statistical Learning (2nd ed.)
• Bailey, D. H. & López de Prado, M.: The Deflated Sharpe Ratio (2014)
• Bailey, Borwein, López de Prado & Zhu: Pseudo-Mathematics and Financial Charlatanism (Notices of the AMS, 2014)
• Harvey, C. R. & Liu, Y.: Backtesting, JPM (2015)

---

6. Connected Graph Bridges

• Back: 01 · From Zero · Index Hub
• Forward: 03 · Multiple Testing → 04 · Deflated Sharpe → 05 · Failure Modes
• Base: Probability & Measure Theory · Sibling: Financial ML Pitfalls