M.5 Statistics & Inference

Basic Prerequisites: Probability & Measure Theory (distributions, expectation, conditional expectation) and Multivariable Calculus & Optimization (Taylor expansions, maximization). (these are the folder-level prerequisites for pages 02–06; page 01 states its own, smaller, entry requirements)

1. Intuition & Practical Objective

Every number a quant reports is an estimate of something unobservable, and every decision (trade, hedge, size a position) is a test whose error rate is a choice. Statistics is the discipline that turns “the sample mean was 8%” into “8% ± 2%, and I would see a number this large by chance $p$ % of the time.” Without it, a backtest is a story; with it, a backtest is an inference with a stated failure probability.

Its claim is sharp: an estimator is a random variable with a distribution, and inference is the study of that distribution. The three primitives are (i) point estimation - how to build and compare estimators (MLE, method of moments) via bias, variance, MSE and the Cramér–Rao bound; (ii) sampling distributions - the Laws of Large Numbers and the Central Limit Theorem that make the sampling distribution of a statistic calculable, and hence the standard error $σ / n$ ; (iii) interval estimation and hypothesis testing - pivots, confidence intervals, size/power, p-values, and the multiplicity corrections that make “the best of $N$ strategies” an honest statement.

This folder is the topic-folder for that toolbox, and it is the statistical backbone of backtesting and model validation: a Sharpe ratio is a $t$ -statistic, a strategy search is a multiple-testing problem, and a model-selection score like AIC is a penalised likelihood.

This page is a hub: it (a) gives the fast estimator/test lookup below (job #1), and (b) routes you to six sub-pages from raw intuition through point estimation, the CLT, intervals & testing, bias–variance & validation, and the bootstrap/asymptotic extensions.

The one-sentence essence. “An estimator is a random variable, the CLT fixes its sampling distribution and hence the standard error $σ / n$ , and inference is just claiming coverage (intervals) or error rates (tests) about that distribution - with multiplicity controlled when you look at many estimates at once.”

2. Mathematical Ground Truth & Derivations

Quick-Reference Lookup (job #1). All formulas below are transcribed from Casella & Berger (2002) Ch 7–9 (primary), cross-checked against ESL Ch 2, 7–8 and Glasserman Ch 1; the numbers in the check column were re-executed and reproduced exactly by the scripts in §3 and on the sub-pages.

Notation: $θ$ parameter, $W = \hat{θ}$ an estimator (a random variable), $E_{θ}$ , $Var_{θ}$ , $ℓ (θ ∣ x) = lo g L (θ ∣ x)$ log-likelihood, $Z \sim N (0, 1)$ , $Φ$ standard normal CDF.

Point estimators and their quality (C&B Ch 7):

Quantity	Formula	Reference	Verified check
MSE decomposition	$E_{θ} (W - θ)^{2} = Var_{θ} W + (Bias_{θ} W)^{2}$ , bias $= E_{θ} W - θ$	Eq 7.3.1	MSE(S²)=3.546 vs MSE(σ̂²_MLE)=3.037, n=10 (matches theory 2σ⁴/(n−1)=3.556, (2n−1)σ⁴/n²=3.040)
MLE (exponential rate)	$\hat{λ} = 1/ \overset{ˉ}{X}$	§7.2.2	$Var (\hat{λ}) = 0.03172$ vs CRLB $λ^{2} / n = 0.03125$
MLE (normal)	$\overset{μ}{^} = \overset{ˉ}{X}, \overset{σ}{^}^{2} = \frac{1}{n} \sum (X_{i} - \overset{ˉ}{X})^{2}$	§7.2.2	biased; smaller MSE than $S^{2}$
Method of moments	equate $E_{θ} [X^{k}]$ to $\frac{1}{n} \sum X_{i}^{k}$ , solve	§7.2.1	shifted-exp shift $\hat{d} = 3.0011$ (true 3.0)
Cramér–Rao lower bound	$Var_{θ} W \geq \frac{[ τ ^{'} ( θ ) ] ^{2}}{n E _{θ} [ ( \frac{\partial}{\partial θ} lo g f ( X ∣ θ ) ) ^{2} ]} = \frac{[ τ ^{'} ( θ ) ] ^{2}}{I _{n} ( θ )}$	Thm 7.3.1	exp rate attains it ( $0.03172 \approx 0.03125$ )
MLE asymptotic variance	$Var (h (\hat{θ})) \approx \frac{[ h ^{'} ( θ ) ] ^{2}}{- ℓ ^{''} ( θ ^ ∣ x )} \sim \frac{[ h ^{'} ( θ ) ] ^{2}}{I _{n} ( θ )}$ (observed info)	Eq 7.4.1	odds-ratio $Var = 0.00177$ vs $\frac{p}{n ( 1 - p ) ^{3}} = 0.00175$
Delta method	$Var g (\overset{ˉ}{X}) \approx [g^{'} (μ)]^{2} Var \overset{ˉ}{X}$ ; multivariate: $\sum_{i} g_{i}^{'2} Var X_{i} + 2 \sum_{i < j} g_{i}^{'} g_{j}^{'} Cov (X_{i}, X_{j})$	Eq 7.4.5	odds-ratio check above
Asymptotic normality of MLE	$\hat{θ} \approx N (θ, \frac{1}{I _{n} ( θ )})$ (regular models, support free of $θ$ )	§7.4.1	-

Sampling distributions, LLN & CLT (Glasserman §1.1; C&B Ch 5):

Result	Statement	Verified check
LLN (SLLN)	$\overset{ˉ}{X}_{n} = \frac{1}{n} \sum X_{i} \to μ$ a.s.	running mean of Exp(1): 1.108→1.0105→0.9968
CLT (Lindeberg–Lévy)	$\frac{X ˉ _{n} - μ}{σ / n} \Rightarrow N (0, 1)$ ; standard error $= σ / n$	sd(mean)=0.3167 vs $1/ 10 = 0.3162$ ; halving error costs 4× data
Rate	$O (n^{- 1/2})$ , independent of dimension (why Monte Carlo wins in high dim)	(D) $sup ∣ F_{n} - Φ∣$ : 0.133( $n = 1$ )→0.037( $n = 16$ )→0.010( $n = 256$ )
Sample variance	$(n - 1) S^{2} / σ^{2} \sim χ_{n - 1}^{2}$	mean $= 10.999$ (theory 11), Var $= 22.03$ (theory 22)
Student- $t$	$(\overset{ˉ}{X} - μ) / (S / n) \sim t_{n - 1}$ (normal data)	$P(

Intervals, tests and multiplicity (C&B Ch 8–9):

Object	Formula	Reference	Verified check
Pivot	$Q (X, θ)$ with distribution independent of $θ$	Def 9.2.1	$(\overset{ˉ}{X} - μ) / (S / n)$ is a pivot
$t$ -confidence interval	$\overset{ˉ}{X} \pm t_{n - 1, α /2} S / n$	Ex 9.2.3	95% coverage: Normal 0.9504, Exponential 0.9123
Approx. CI (CLT)	$W \pm z_{α /2} Var (W)$	§9.4.2	-
Neyman–Pearson	most powerful level- $α$ test of simple vs simple rejects where likelihood ratio $L (θ_{1}) / L (θ_{0}) > k$	Thm 8.3.1	-
Size / power	size $α = sup_{θ \in Θ_{0}} P_{θ} (reject)$ ; power $= P_{θ} (reject)$ , $θ \in Θ_{1}$	§8.3.1	Bonferroni keeps FWER $\leq 0.05$ : 0.0483
p-value	smallest $α$ at which the sample is rejected (data-dependent)	§8.3.3	-
Wilks (LRT)	$- 2 lo g Λ \to χ_{ν}^{2}$ , $ν = dim Θ - dim Θ_{0}$	Thm 8.4.1	Poisson mean LRT: mean 0.9941 (χ²₁ mean 1), 95th pct 3.861 vs 3.842
Multiple testing	Bonferroni $α / m$ ; FDR (Benjamini–Hochberg) controls $E [V / max (R, 1)]$	-	20 nulls: P(≥1 raw $p < 0.05$ )=0.647 (theory 0.642)
Model selection	AIC $= - 2 lo g L + 2 k$ (ESL $- \frac{2}{N} lo g L + \frac{2 d}{N}$ ); BIC $= - 2 lo g L + k lo g n$	ESL 7.29/7.35; Tsay 2.16	AIC/BIC both pick degree 1

3. Computational Implementation - MSE, Cramér–Rao and the CLT in one script

Standard library only. Verifies the MSE decomposition, that the exponential-rate MLE attains the Cramér–Rao bound, and the $σ / n$ standard error.

4. Failure Modes & First-Principles Breakdowns

Hub signposts - the folder’s failure-mode analysis lives on the sub-pages. In one line each:

Estimating a ratio/nonlinear function and trusting the plug-in variance. When $h (θ)$ is non-monotone the delta method can report (near-)zero variance - the MLE of $p (1 - p)$ gives $Var = 0$ at $p = \frac{1}{2}$ (C&B §7.4.1). Non-monotone transforms break the approximation.
Using the CLT when the sample is too small or too skewed. $t$ -intervals on Exponential(1) data under-cover (0.912 not 0.950); the CLT error is $O (skew / n)$ .
Testing many hypotheses and reporting the winner. With $m = 20$ true nulls, P(at least one $p < 0.05$ ) $= 0.647$ ; with $N$ strategies the best spurious $t \approx 2 lo g N$ - “significant” at $1.96$ is guaranteed noise. Correct with Bonferroni/FDR or a Reality-Check bootstrap.
Validating on data you already used. Selecting features/models on the full sample then cross-validating leaks information: the ESL screening example reports CV error 3% against a true 50% (ESL §7.10.2). Screening must happen inside each fold.
Confusing bias with error. An unbiased estimator need not be closest (MSE = Var + Bias²); the biased $\overset{σ}{^}_{MLE}^{2}$ beats unbiased $S^{2}$ on MSE.

5. References

Casella, G. & Berger, R. L.: Statistical Inference (2nd ed., 2002)
Hastie, Tibshirani & Friedman: Elements of Statistical Learning (2nd ed., 2009)
Glasserman, P.: Monte Carlo Methods in Financial Engineering (2004)
Tsay, R. S.: Analysis of Financial Time Series (3rd ed., 2010)
Efron, B. & Tibshirani, R.: An Introduction to the Bootstrap (1993)

6. Connected Graph Bridges

Foundational base: Probability & Measure Theory · Calculus & Optimization
Sibling foundations: Econometrics & Time Series (stationarity & MLE for GARCH) · Ergodicity & Statistical Mechanics (time vs ensemble averages) · Numerical Methods (Monte Carlo uses LLN/CLT)
Sub-pages (in-folder): 01 From Zero · 02 Point Estimation · 03 The CLT & Sampling · 04 Confidence Intervals & Testing · 05 Bias–Variance & Validation · 06 Advanced Extensions

Beginner: start at 01 · Practitioner: start at 05

Kwant Atlas

Explorer

M.5 Statistics & Inference

1. Intuition & Practical Objective

2. Mathematical Ground Truth & Derivations

3. Computational Implementation - MSE, Cramér–Rao and the CLT in one script

4. Failure Modes & First-Principles Breakdowns

5. References

6. Connected Graph Bridges

M.5.1 Statistics & Inference from Zero

M.5.2 Point Estimation

M.5.3 The CLT & Sampling

M.5.4 Confidence Intervals & Hypothesis Testing

M.5.5 Bias–Variance, Cross-Validation & Model Selection

M.5.6 Advanced Extensions