M.5.4 Confidence Intervals & Hypothesis Testing

Basic Prerequisites: 02 · Point Estimation and 03 · The CLT & Sampling.

---

1. Intuition & Practical Objective

A point estimate is half an answer. The other half is a set of plausible values (a confidence interval) or a decision with a stated error rate (a hypothesis test). These two are the same construction viewed from two sides: a $1-\alpha$ confidence interval is exactly the set of parameter values that a level-$\alpha$ test would not reject (C&B Thm 9.2.1). Inverting the test gives the interval; the interval’s coverage gives the test’s size.

For a quant this is not academic:

• “The strategy’s mean return is positive” $\Rightarrow$ a one-sided $t$-test, and the Sharpe ratio’s $t$-statistic $=\text{SR}\sqrt{T}$.
• “The fund’s VaR is 2%” $\Rightarrow$ an interval estimator of a tail quantile.
• A p-value is the probability, under the null, of seeing a test statistic at least as extreme as observed - not the probability the null is true (the single most abused number in finance).

Why the grader cares. Backtest overfitting is a multiplicity problem: test $N$ strategies and the best looks significant by construction; the expected best spurious $t$-statistic grows like $\sqrt{2\log N}$. Testing theory, done right, is what separates a real signal from the best of $N$ coin flips.

---

2. Mathematical Ground Truth & Derivations

2.1 Intervals by inverting tests (C&B §9.2.1)

For each ${\theta }_0$, let $A({\theta }_0)$ be the acceptance region of a level-$\alpha$ test of $H_0:\theta ={\theta }_0$. Then

$$C(x)=\{{\theta }_0:x\in A({\theta }_0)\}$$

is a $1-\alpha$ confidence set; its coverage probability satisfies $P_{\theta }(\theta \in C(X))\ge 1-\alpha$. The confidence coefficient is ${\inf }_{\theta }{P}_{\theta }(\theta \in C(X))$ (C&B §9.3.1).

2.2 Pivots (C&B §9.2.2)

A pivot is $Q(X,\theta )$ whose distribution is independent of all parameters (Def 9.2.1). For normal data, $T=(\bar{X}-\mu )/(S/\sqrt{n})\sim t_{n-1}$ is a pivot, giving the exact $t$-interval

$$\bar{X}\pm t_{n-1,\alpha /2}\frac{S}{\sqrt{n}}.$$

When no pivot exists, the CLT gives an approximate interval (C&B §9.4.2): for a statistic $W$ with $\widehat{\mathrm{Var}}(W)$,

$$W\pm z_{\alpha /2}\sqrt{\widehat{\mathrm{Var}}(W)}.$$

For MLEs the likelihood-based interval inverts the (asymptotically normal) pivot $Q(X,\theta )=\frac{{\partial }_{\theta }\log L(\theta \mid X)}{\sqrt{-I_n(\theta )}}\sim N(0,1)$ (eq 9.4.1).

2.3 Hypothesis testing: size, power, p-value (C&B §8.3)

Reject if the test statistic falls in the rejection region $R$. Two error types:

• Type I ($H_0$ true, rejected), probability $\alpha =P_{\theta \in {\Theta }_0}(X\in R)$ - the size $\alpha ={\sup }_{\theta \in {\Theta }_0}{P}_{\theta }(X\in R)$.
• Type II ($H_1$ true, not rejected); power $\beta (\theta )=P_{\theta }(X\in R)$, $\theta \in {\Theta }_1$. Ideal: power 0 on ${\Theta }_0$, 1 on ${\Theta }_1$.

The p-value is the smallest $\alpha$ at which the observed data would be rejected; equivalently $p={\sup }_{\theta \in {\Theta }_0}{P}_{\theta }(T\ge T_{\text{obs}})$ - data-dependent, not a fixed error rate (C&B §8.3.3).

2.4 Optimal tests (C&B §8.3.2, §8.2.1)

• Neyman–Pearson Lemma (Thm 8.3.1): for simple $H_0:\theta ={\theta }_0$ vs $H_1:\theta ={\theta }_1$, the most powerful level-$\alpha$ test rejects $H_0$ when the likelihood ratio $L({\theta }_1)/L({\theta }_0)>k$, with $k$ set by the size.
• Karlin–Rubin (Thm 8.3.2): for a monotone likelihood-ratio family, a threshold test on the sufficient statistic is UMP for one-sided hypotheses.
• Generalised likelihood-ratio test (Def 8.2.1): reject for small $\Lambda (x)=\frac{{\sup }_{\theta \in {\Theta }_0}L(\theta \mid x)}{{\sup }_{\theta \in \Theta }L(\theta \mid x)}$. Wilks’ theorem (Thm 8.4.1): under $H_0$, $-2\log \Lambda \Rightarrow {\chi }_{\nu }^2$, $\nu =\dim \Theta -\dim {\Theta }_0$.

2.5 Multiplicity (the finance-critical part)

Testing $m$ hypotheses at level $\alpha$ each inflates the family-wise error rate (FWER):

$$\text{FWER}=P(\ge 1\text{ false rejection})\le 1-(1-\alpha {)}^m\approx m\alpha \text{(independence, small }\alpha ).$$

• Bonferroni: use $\alpha /m$ per test $\Rightarrow$ FWER $\le \alpha$.
• Benjamini–Hochberg FDR: sort p-values $p_{(1)}\le \cdots \le p_{(m)}$, reject the largest $k$ with $p_{(k)}\le \frac{k}{m}q$; controls the expected proportion of false discoveries $\mathbb{E}[V/\max (R,1)]\le q$.
• Max-statistic correction: the best of $N$ independent zero-skill strategies has $t$-statistic with $\mathbb{E}[\max ]\approx \sqrt{2\log N}$ - the deflated Sharpe / White Reality Check threshold.

---

3. Computational Implementation - t-Intervals, LRT, and the Multiplicity Tax

Standard library only, with a self-contained regularised incomplete gamma (for ${\chi }^2$) and normal inverse. We check $t$-interval coverage, a Poisson likelihood-ratio test against ${\chi }_1^2$, and the multiplicity problem.

The $t$-interval is exact for normal data (0.9504) but under-covers on exponential data (0.9123) - the CLT’s finite-sample error, visible. The LRT statistic matches its ${\chi }_1^2$ limit. And 20 independent nulls produce at least one “significant” result 65% of the time - Bonferroni restores the 5% FWER (0.0483).

---

4. Failure Modes & First-Principles Breakdowns

1. Interpreting $p=0.04$ as “4% chance the null is true”. The p-value is $P(\text{data this extreme}\mid H_0)$, not $P(H_0\mid \text{data})$; the two differ by the prior odds (Bayes’ rule). A 4% p-value on a coin that all prior evidence says is fair is weak evidence.
2. Ignoring multiplicity. Reporting the best of many tested strategies at a raw 5% level is a near-certain false positive; the expected best-of-$N$ spurious $t\approx \sqrt{2\log N}$. Correct with Bonferroni/FDR or a bootstrap Reality Check.
3. Confusing statistical and economic significance. A t-stat of 3 from a century of daily data with a 1-basis-point edge is statistically “real” and economically worthless; and a short high-Sharpe sample can be economically huge and statistically indistinguishable from noise.
4. Under-covering intervals. Using $z$ instead of $t$, or the CLT on short fat-tailed samples, produces intervals that cover less than advertised (0.912 not 0.950) - your “95%” is really 91%.
5. One-sided vs two-sided silent switch. Peeking at the data, then choosing the direction of a one-sided test, doubles the true Type-I rate. Pre-specify the test.
6. The LRT’s edge-of-support trap. Wilks’ theorem fails when the null is on the boundary of the parameter space or the support depends on $\theta$; then $-2\log \Lambda$ is not ${\chi }_{\nu }^2$. Check the regularity conditions.

---

5. References

• Casella & Berger, Statistical Inference
• Hastie, Tibshirani & Friedman, ESL
• White, H. (2000), A Reality Check for Data Snooping
• Harvey, Liu & Zhu (2016), “…and the Cross-Section of Expected Returns”

---

6. Connected Graph Bridges

• Back: 03 · The CLT & Sampling · 02 · Point Estimation · Index Hub
• Continue: 05 · Bias–Variance & Validation · 06 · Advanced Extensions
• Applications: Quantitative Research (backtesting, signal validation) · VaR & Expected Shortfall (tail interval estimation) · Econometrics (ACF/PACF and Ljung–Box are hypothesis tests)