1.2 Backtesting Hygiene

Basic Prerequisites: Probability & Measure Theory (order statistics, extreme-value theory, hypothesis testing). (these are the folder-level prerequisites for pages 02–06; page 01 states its own, smaller, entry requirements)

1. Intuition & Practical Objective

A backtest is not a measurement - it is the maximum of a search. The moment a researcher runs a parameter sweep (holding periods, entry thresholds, stop losses, universes, model families) and reports the winner, the reported statistic is a maximum of $N$ random variables, not an unbiased estimate of one. Under the null of zero skill, that maximum still grows without bound as $N \to \infty$ . So the number of trials $N$ is not a footnote: it is the single most important number missing from almost every published backtest, and without it a Sharpe ratio is uninterpretable.

This folder is the topic-hub for backtesting hygiene in Kwant-Atlas. It (a) gives you the fast formula/decision lookup below - the job #1 of a hub - and (b) routes you to six sub-pages that walk from raw intuition through the multiple-testing problem, the Deflated Sharpe Ratio (DSR), practice, and advanced tooling.

The one-sentence essence. “A reported Sharpe ratio overstates skill by exactly the amount by which the maximum of $N$ null trials exceeds zero - so deflate it by the order statistic $E [max_{n} SR_{n}]$ , adjusted for sample length and non-normal returns.”

2. Mathematical Ground Truth & Derivations

Quick-Reference Lookup (job #1). Notation: $SR$ = estimated (non-annualized, per-period) Sharpe ratio, $N$ = number of independent trials, $T$ = number of observations, $\overset{γ}{^}_{3}, \overset{γ}{^}_{4}$ = sample skewness and (Pearson) kurtosis of the selected strategy’s returns, $Φ$ = standard normal CDF, $γ \approx 0.5772$ the Euler–Mascheroni constant. All formulas below were re-executed and reproduced numerically (see §3).

Quantity	Formula	Verified check
Family-wise error ( $N$ tests, level $α$ )	$FWER = 1 - (1 - α)^{N} \approx N α$	$α = 0.05, N = 100 \Rightarrow 0.9941$
Expected max SR (EVT approx, per unit $σ$ )	$E [max] \approx 2 ln N + \frac{γ}{2 ln N}$	$N = 100 \Rightarrow 3.2250$
Expected max SR (exact order statistic, per $σ$ )	$E [max] = (1 - γ) Φ^{- 1} (1 - \frac{1}{N}) + γ Φ^{- 1} (1 - \frac{1}{N e})$	$N = 100 \Rightarrow 2.5306$ ; $N = 1000 \Rightarrow 3.2551$
Selection threshold (null, in SR units)	$SR_{0} = V [{SR_{n}}] \cdot [(1 - γ) Φ^{- 1} (1 - \frac{1}{N}) + γ Φ^{- 1} (1 - \frac{1}{N e})]$	example: $SR_{0}^{(ann)} = 1.7894$
Deflated Sharpe Ratio	$DSR = Φ \frac{( SR - SR _{0} ) T - 1}{1 - γ ^ _{3} SR + \frac{γ ^ _{4} - 1}{4} SR ^{2}}$	example: $DSR (N = 100) = 0.8997$
Minimum Track Record Length (PSR $> 0.95$ )	$MinTRL = 1 + [1 - \overset{γ}{^}_{3} SR + \frac{γ ^ _{4} - 1}{4} SR^{2}] (\frac{Φ ^{- 1} ( 0.95 )}{SR})^{2}$	$SR = 2.5$ ann (sk $- 3$ , kurt $10$ ) $\Rightarrow 168$ days
Harvey–Liu multiple-testing $p$ -value	$p_{M} = 1 - (1 - p_{S})^{N}$ ; haircut SR found by inverting $p_{M}$	$p_{S} = 0.0008, N = 200 \Rightarrow p_{M} = 0.1473$ ; HSR $= 0.3241$ (haircut $57%$ )
Bonferroni / Holm (FWER)	$p_{(i)}^{Bonf} = min {N p_{(i)}, 1}$ ; Holm adjusts sequentially	-
Probability of Backtest Overfitting	$PBO = Pr [λ < 0]$ , $λ_{c} = ln \frac{ω ˉ _{c}}{1 - ω ˉ _{c}}$ (CSCV logits)	pure noise $N = 50 \Rightarrow 0.504$

Critical caveat (Bailey & López de Prado §A.3). $N$ is the number of independent trials. If you ran $M$ correlated trials, use the implied count $N \approx ρ (M - 1) + 1$ ( $ρ$ = average off-diagonal correlation). Plugging raw $M$ overstates the threshold and over-penalizes; ignoring correlation understates it. Also: DSR $< 0.95$ is not proof of fraud - it is the statement that the observed Sharpe is not distinguishable from the best of $N$ coin flips at the 95% level.

3. Computational Implementation - the hygiene engine

Standard-library only. It reproduces every verified number in §2: the expected-max-Sharpe order statistic, the DSR of the canonical Bailey–López de Prado example, and the MC check that the order statistic beats the crude EVT approximation.

Read the output as the paper’s punchline: the strategist should have been fundable after only $N = 46$ independent trials - but ran 100, so the same $SR = 2.5$ deflates to $DSR = 0.900$ , below the 95% bar. Non-normality (skew $- 3$ , kurtosis $10$ ) is a second, independent source of inflation: had the returns been normal, the strategy would have survived up to $N \approx 88$ trials.

4. Failure Modes & First-Principles Breakdowns

Hub signposts - the folder’s practice checklist lives in 05 · Failure Modes & Practice. In one line each:

The missing $N$ . A backtest that does not disclose the number of trials is uninterpretable (“worthless”, B&LdP §abstract) - the reported Sharpe is the max of an unknown number of draws.
Holdout ≠ hygiene. A single train/test split assesses generality as if one trial occurred; apply holdout 20 times and a false positive at 5% becomes expected, not unlikely.
Independence is a fiction. Correlated trials make the effective $N$ far smaller than the raw sweep count; ignoring this both mis-calibrates DSR and hides that your “1000 ideas” were really ~10.
Leakage across folds. Standard $k$ -fold CV on financial data leaks serial-correlation/label overlap between train and test; you need purging + embargo (page 06).
Memory effects make it worse than neutral. In mean-reverting series, overfit rules undo their in-sample patterns out of sample - overfitting becomes loss-maximization, not merely zero-edge.

5. References

Bailey, David H. & López de Prado, Marcos: The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality, Journal of Portfolio Management 40(5), 94–107 (2014). The formula-authoritative source for this folder; the worked example above is reproduced exactly.
Harvey, Campbell R. & Liu, Yan: Backtesting, Journal of Portfolio Management 42(1), 13–28 (2015). The multiple-testing / haircut Sharpe framework (FWER, FDR, Bonferroni–Holm).
Bailey, Borwein, López de Prado & Zhu: The Probability of Backtest Overfitting, Journal of Computational Finance 20(4) (2017). CSCV and the PBO statistic.
White, Halbert: A Reality Check for Data Snooping, Econometrica 68(5), 1097–1126 (2000). The bootstrap/reality-check test that the best model in a search has no predictive edge over a benchmark.
Hastie, Tibshirani & Friedman: The Elements of Statistical Learning (2nd ed., 2009)

6. Connected Graph Bridges

Foundational base: Probability & Measure Theory · order statistics & extreme-value theory
Sibling topics: Feature Engineering & Labeling (triple-barrier labels feed the purge) · Stat-Arb & Pairs Trading
Cross-pillar: Financial ML Pitfalls & Low SNR (the same overfitting disease, ML flavour)
Sub-pages (in-folder): 01 From Zero · 02 Why Backtests Lie · 03 Multiple Testing · 04 Deflated Sharpe · 05 Failure Modes · 06 Advanced Extensions

Beginner: start at 01 · Practitioner: start at 05

Kwant Atlas

Explorer

1.2 Backtesting Hygiene

1. Intuition & Practical Objective

2. Mathematical Ground Truth & Derivations

3. Computational Implementation - the hygiene engine

4. Failure Modes & First-Principles Breakdowns

5. References

6. Connected Graph Bridges

1.2.1 Backtesting Hygiene from Zero

1.2.2 Why Backtests Lie

1.2.3 The Multiple-Testing Problem

1.2.4 The Deflated Sharpe Ratio

1.2.5 Failure Modes & Real-World Practice

1.2.6 Advanced Extensions