1.2.3 The Multiple-Testing Problem

Basic Prerequisites: Probability & Measure Theory (order statistics, extreme-value distributions) and 01 · From Zero.

---

1. Intuition & Practical Objective

Test one hypothesis at the 5% level and you have a 5% chance of a false positive. Test 100 and the chance that at least one is a false positive - under the null of no skill anywhere - is 99.4%. This is the multiple-testing problem, and it is not a subtle statistical caveat: it is the arithmetic reason a large theory-free parameter sweep is guaranteed to “find” something.

The practical objective of this page is to put a number on the bar a backtest must clear once you admit how many trials it survived. Two equivalent framings:

• Extreme-value framing (Bailey & López de Prado): the reported Sharpe is ${\max }_n{\widehat{SR}}_n$; its expected value under the null is a known order statistic of $N$ draws. (→ feeds the DSR, page 04.)
• $p$-value framing (Harvey & Liu / White): the single-test $p$-value $p_S$ must be inflated to a multiple-testing $p_M$ that reflects the search. (→ the haircut Sharpe.)

Both answer “how high does the bar go?”, and they agree in spirit: the bar rises with $N$, slowly (logarithmically) but without limit.

---

2. Mathematical Ground Truth & Derivations

2.1 Family-wise error rate (FWER)

With $N$ independent tests at level $\alpha$,

$$\text{FWER}=1-(1-\alpha {)}^N\approx N\alpha (\alpha \text{small}).$$

So $\alpha =0.05$: $N=10\Rightarrow 0.401$, $N=100\Rightarrow 0.994$, $N=1000\Rightarrow 1.000$. Bonferroni controls FWER by inflating each $p$: $p_{(i)}^{\text{Bonf}}=\min \{N{p}_{(i)},1\}$; Holm applies the same idea sequentially, $p_{(i)}^{\text{Holm}}=\min \{1,{\max }_{j\le i}[(N-j+1)p_{(j)}]\}$, and is uniformly more powerful. For thousands of tests FWER is too conservative, and the false-discovery-rate (Benjamini–Hochberg) controls instead the proportion of false rejections among the rejected (Harvey & Liu §“multiple testing framework”).

2.2 Expected maximum Sharpe under $N$ null trials

Let $\{{\widehat{SR}}_n\}$ be $N$ i.i.d. trial Sharpe estimates with mean $\mu$ and variance $V[\{{\widehat{SR}}_n\}]={\sigma }^2$ (in per-period units). Standardize: $Z_n=({\widehat{SR}}_n-\mu )/\sigma$. The expected maximum of $N$ standard normals has two standard forms:

EVT approximation (loose, ubiquitous):

$$\mathbb{E}\left[\underset{n}{\max }{Z}_n\right]\approx \sqrt{2\ln N}+\frac{\gamma }{\sqrt{2\ln N}},\gamma \approx \mathrm{0.5772.}$$

Exact order-statistic form (used by the DSR):

$$\mathbb{E}\left[\underset{n}{\max }{Z}_n\right]=(1-\gamma ){\Phi }^{-1}(1-\frac{1}{N})+\gamma {\Phi }^{-1}(1-\frac{1}{Ne}).$$

The selection threshold in Sharpe units is then

$${\widehat{SR}}_0=\mu +\sigma \cdot \mathbb{E}\left[\underset{n}{\max }{Z}_n\right],\text{under }{H}_0:\mu =0\Rightarrow {\widehat{SR}}_0=\sqrt{V[\{{\widehat{SR}}_n\}]}\cdot \mathbb{E}\left[\underset{n}{\max }{Z}_n\right].$$

The approximation is not innocent. The crude EVT form overstates the true expected maximum by ~28% at $N=100$ (3.225 vs 2.531) - using it makes you over-penalise a good strategy. Monte Carlo settles which is right (see §3). The order-statistic version is the one to use.

2.3 The Harvey–Liu haircut

Transform Sharpe to a $t$-statistic, $t=\widehat{SR}\cdot \sqrt{T}$ (net of any factor adjustment, $SR$ non-annualized). Compute the single-test two-sided $p$-value $p_S=\mathrm{Pr}(|r_{T-1}|>t)$. Under $N$ independent tests the multiple-testing $p$-value is

$$p_M=1-(1-p_S{)}^N.$$

The haircut Sharpe $HSR$ is found by equating a single test’s $p$-value to $p_M$: solve $\mathrm{Pr}(|r_{T-1}|>t^{\ast })=p_M$ and set $HSR=t^{\ast }/\sqrt{T}$. Because $p_M\gg p_S$, $HSR\ll \widehat{SR}$. Harvey & Liu’s rule: the haircut is non-linear - very high Sharpes are barely penalised, marginal ones are heavily penalised (a flat 50% is wrong in both directions).

2.4 How many trials are optimal? The 1/e-law

Since each added trial raises the false-positive floor, there is an optimal stopping rule (the “secretary problem”, Bruss 1984): sample a random $\frac{1}{e}\approx 37\%$ of the theory-justified configurations, measure them, then keep drawing one at a time until one beats all previous - take it. Every further trial after that is pure overfitting. Theory, not compute, should choose the search.

2.5 Independent vs effective $N$

$N$ is the count of independent trials. If you ran $M$ correlated trials with average off-diagonal correlation $\hat{\rho }$, the implied independent count is

$$\hat{N}\approx \hat{\rho }(M-1)+1.$$

Using raw $M$ overstates the threshold; treating correlated trials as independent understates it. Report the full sweep $M$ and an estimate of $\hat{\rho }$ - a strategy found among 1,000 correlated variants is often really ~10 independent bets (Bailey & López de Prado §A.3).

---

3. Computational Implementation - the three numbers

Stdlib only. (1) The expected-maximum table with Monte Carlo verification, (2) the Harvey–Liu haircut, (3) FWER and Bonferroni.

Three readings: (1) the exact order statistic tracks Monte Carlo (2.531 vs 2.507 at $N=100$) while the EVT approximation overshoots (3.225) - use the exact form. (2) An honest-looking annual Sharpe of 0.75 over 20 years is cut to 0.32 (a 57% haircut) once $N=200$ trials are acknowledged - closely matching the “50% discount” folklore, but derived rather than assumed. (3) Bonferroni’s adjusted $p$-values reproduce Harvey & Liu’s textbook sequence exactly (only the first of six “discoveries” survives), and FWER at $N=100$ is 99.4%.

---

4. Failure Modes & First-Principles Breakdowns

1. Counting trials you can remember. The denominator $N$ must include every configuration evaluated, including the exploratory ones you “weren’t really serious about”. Pre-register the search to know $N$.
2. Treating correlated trials as independent. Position sizing, stops, and filters applied onto one base signal produce thousands of dependent trials. Use $\hat{N}=\hat{\rho }(M-1)+1$.
3. Using the EVT approximation. $\sqrt{2\ln N}+\gamma /\sqrt{2\ln N}$ over-penalizes by ~28% at $N=100$. For DSR the exact order statistic is the correct bar.
4. FWER when you don’t need it. With thousands of tests, Bonferroni virtually never rejects; FDR (Benjamini–Hochberg) is the appropriate error rate and is what Harvey–Liu advocate for factor zoos.
5. Treating $p_M$ as a final verdict. A non-rejection is not proof of falsehood; it is the statement that the result is not distinguishable from the best of $N$ flukes. Combine with DSR, PBO, and economics.
6. Endless testing. Once the optimal-stopping point is passed, additional trials guarantee inflation - the secretary-problem $1/e$ rule is a hard discipline, not a suggestion.

---

5. References

• Bailey, D. H. & López de Prado, M.: The Deflated Sharpe Ratio (2014)
• Harvey, C. R. & Liu, Y.: Backtesting, JPM (2015)
• White, H.: A Reality Check for Data Snooping, Econometrica 68(5) (2000)
• Bruss, F. T.: A Unified Approach to a Class of Best Choice Problems, Annals of Probability 12(3) (1984)
• Bonferroni / Holm / Benjamini–Hochberg

---

6. Connected Graph Bridges

• Back: 02 · Why Backtests Lie · Index Hub
• Forward: 04 · The Deflated Sharpe Ratio (the order statistic becomes the DSR threshold) → 06 · PBO & Reality Check
• Base: Probability & Measure Theory · Cross-pillar: Financial ML Pitfalls