Basic Prerequisites: none beyond high-school probability. A willingness to distrust a good-looking equity curve is required.
1. Intuition & Practical Objective
This page builds the why of backtesting hygiene with no statistics background needed. The objective is one idea: a backtest reports the best result of a search, and the best of many random results always looks good - so a backtest is evidence of nothing until you say how hard you searched.
Start with the dumbest question: what is a backtest? Naively, “a historical simulation of how a strategy would have performed.” But nobody runs exactly one backtest. You tried a 20-day moving average, then 50, then 200; you added a stop loss, removed it; you restricted the universe; you flipped the sign. Your final report contains the survivor of that whole process. The number you printed is not the strategy’s return - it is the maximum of a set of returns you generated by fishing.
Three steps:
-
Random data always contains patterns. A fair coin flipped 1,000 times contains runs, streaks, and “seasonality” - somewhere. A researcher who scans enough of these patterns finds a rule that “predicts” them. The rule has zero power on the next 1,000 flips, because there was never a signal to find.
-
The winner is selected for its luck. Suppose 1,000 strategies have zero true skill. Their estimated Sharpe ratios are noisy draws around zero. The best of 1,000 noisy draws is not near zero - it is a large positive number, purely by the arithmetic of taking a maximum. That is the winner’s curse: the strategies you select are disproportionately the ones whose noise happened to be favorable.
-
You only see the survivors. Researchers, journals, and fund databases publish winners. The 999 losers are invisible (“file drawer”). So the observed distribution of published backtests is itself a selection - biased exactly toward overstatement.
The one-sentence essence. “A backtest is the maximum of trials; unless you account for and the chance-driven height of that maximum, a beautiful equity curve is indistinguishable from 1,000 coin flips.”
2. Mathematical Ground Truth & Derivations
The coin-flip demonstration (Bailey & López de Prado 2014). Toss a fair coin ten times. A sequence like appears roughly once every attempts. A researcher declares the optimal betting rule: “bet heads for the first five tosses, tails for the last five.” That is a seasonal strategy. On the next ten tosses () it wins 5 and loses 5 - the rule was fit to a random pattern that did not recur. Nothing broke; there was never a signal. This is backtest overfitting in its purest form.
Where the exaggeration comes from - arithmetic, not psychology. Let the trials’ estimated Sharpe ratios be , drawn around a true mean of zero (no skill) with cross-trial variance . The reported statistic is . Order statistics tell us its expected value is positive and increasing in , even though the truth is zero:
The precise size of that drift is the subject of 03 · The Multiple-Testing Problem, and the deflation it demands is 04 · The Deflated Sharpe Ratio.
In-sample vs out-of-sample - the two words that carry all the weight. Split history into an in-sample (IS) part you learn on and an out-of-sample (OOS) part you validate on. The gap between IS and OOS performance is the performance degradation, and in a real series with memory it is typically negative, not merely zero (a fitted pattern gets actively “undone”).
Why “I only backtested once” is almost never true. Every choice - universe, dates, cost model, rebalance frequency, data vendor - was tried in some form. The honest question is never “did I overfit?” (yes, somewhat) but “how many effective independent trials did this result survive?“
3. Computational Implementation - watch a coin become a “strategy”
Two stdlib experiments. Experiment A shows a pure-noise rule scoring a perfect-looking win-rate; Experiment B shows the best of zero-skill strategies growing with . Both use only random, so the mechanism is fully transparent.
Two messages. A: with 1,000 searches on short samples, the best of pure coin flips “wins” 85% of the time - the “strategy” is a coin (with 1,000 flips each, the law of large numbers dilutes the fluke back toward 0.5, but real backtests are short relative to their search space, so the short-sample regime is the realistic one). B: with zero skill and only 3 years of daily data, the best of 1,000 strategies still shows an annualized Sharpe - and the exact order-statistic formula predicts it (1.879 vs 1.882). There is no signal; the Sharpe is the height of a maximum.
4. Failure Modes & First-Principles Breakdowns
- “But my strategy makes economic sense.” A story attached to a pattern does not turn the pattern into a signal. Narrative and multiple-testing are orthogonal: a genuinely motivated hypothesis still has to clear a threshold that accounts for how the data were searched. (Motivation reduces the effective ; it does not set it to one.)
- Confusing IS with OOS. If you tune any parameter on the OOS window - even “just the stop loss” - it is no longer out of sample. The single most common self-deception.
- Survivorship in the sample itself. Backtesting on today’s index constituents tells you how today’s winners behaved after being selected as winners. This is a silent upward bias that has nothing to do with multiple testing.
- The “one trial” fallacy. Believing a single holdout validates a strategy ignores that the holdout was itself reached after many discarded attempts - see 02 · Why Backtests Lie and the holdout critique on page 03.
5. References
- Bailey, D. H. & López de Prado, M.: The Deflated Sharpe Ratio (2014)
- White, H.: A Reality Check for Data Snooping, Econometrica (2000)
- Hastie, Tibshirani & Friedman: The Elements of Statistical Learning (2nd ed.)
6. Connected Graph Bridges
- Base: Probability & Measure Theory (same-maximum-of-noisy-draws logic beneath order statistics).
- Continue: 02 · Why Backtests Lie · 03 · Multiple Testing · Index Hub.
- Sibling: Financial ML Pitfalls & Low SNR.