M.5.1 Statistics & Inference from Zero

Basic Prerequisites: Elementary algebra and the idea of an average. (No prior probability needed; the sibling Probability from Zero builds the formal base.)

---

1. Intuition & Practical Objective

This page builds the why of statistics with no prior statistics needed. The objective is one idea: an estimate is not a number - it is a random variable, and the only honest way to report it is with its distribution.

Start with the dumbest question: I computed the average daily return of a strategy from three years of data. Why isn’t that the answer? Because a different three years would have given a different average. Your data is one draw from a random process; the average you computed is a function of a random sample, hence itself random. Statistics is the study of how that randomness behaves - how far the estimate typically lands from the truth, and how that distance shrinks as you collect more data.

Three steps:

1. An estimator is a random variable; the truth is a fixed (unknown) constant. Write $\hat{\theta }=W(X_1,\dots ,X_n)$ for an estimate and $\theta$ for the parameter it targets. $\hat{\theta }$ has a sampling distribution - the distribution of values it takes across hypothetical repeated samples. “Confidence” is a property of that distribution, not of any single dataset.

2. More data makes the estimate concentrate, at rate $1/\sqrt{n}$. The Law of Large Numbers says the estimate converges to $\theta$; the Central Limit Theorem says the fluctuation around $\theta$ is approximately normal with standard deviation $\sigma /\sqrt{n}$ - the standard error. This single fact is the reason halving your error requires four times the data, and the reason backtests need long histories.

3. Averages are stable because of cancellation, not because the data is nice. Even from a wildly skewed population (an exponential, say), the sample mean becomes bell-shaped as $n$ grows. The individual observations stay skewed; their average does not. That emergence of normality from non-normal data is the CLT, and it is the engine behind nearly every interval and test in this folder.

The one-sentence essence. “Your estimate is a random variable; the CLT tells you its distribution is normal with spread $\sigma /\sqrt{n}$; everything else - intervals, tests, p-values - is a consequence of that one fact.”

---

2. Mathematical Ground Truth & Derivations

The sampling distribution, formally. Let $X_1,\dots ,X_n$ be i.i.d. with mean $\mu$ and variance ${\sigma }^2$. The sample mean is

$${\bar{X}}_n=\frac{1}{n}\sum _{i=1}^n{X}_i,\mathbb{E}[{\bar{X}}_n]=\mu ,\mathrm{Var}({\bar{X}}_n)=\frac{{\sigma }^2}{n}.$$

So the mean is unbiased with variance falling like $1/n$ - the standard deviation falls like $1/\sqrt{n}$.

Law of Large Numbers (SLLN). ${\bar{X}}_n\to \mu$ almost surely: the estimate concentrates on the truth (Glasserman §1.1).

Central Limit Theorem (Lindeberg–Lévy). Standardising,

$$\frac{{\bar{X}}_n-\mu }{\sigma /\sqrt{n}}⟹N(0,1)\text{as }n\to \infty ,$$

so for large $n$, ${\bar{X}}_n\approx N\left(\mu ,{\sigma }^2/n\right)$. The standard error is $\mathrm{SE}=\sigma /\sqrt{n}$, estimated by $s/\sqrt{n}$ with $s^2=\frac{1}{n-1}\sum (X_i-\bar{X}{)}^2$.

Rewriting the sums (why this is linear, not magic). For i.i.d. data ${\bar{X}}_n-\mu =\frac{1}{n}\sum (X_i-\mu )$ is a sum of $n$ independent zero-mean pieces each of order $\pm \sigma$; the sum has spread $\sqrt{n}\sigma$, and dividing by $n$ gives spread $\sigma /\sqrt{n}$. That is the entire derivation: independent errors add in quadrature, then get divided by $n$.

Convergence rate of the distribution (Berry–Esseen). The CLT approximation error obeys ${\sup }_x|P(({\bar{X}}_n-\mu )/(\sigma /\sqrt{n})\le x)-\Phi (x)|\le C\rho /\sqrt{n}$, where $\rho =\mathbb{E}|X-\mu {|}^3/{\sigma }^3$ is the skewness-like third moment. Fat-tailed or highly skewed populations converge more slowly - the seed of every small-sample CLT failure.

---

3. Computational Implementation - the sampling distribution come alive

Standard library only. We draw from a skewed Exponential(1) population (mean 1, sd 1, skew 2) and watch the sampling distribution of the mean turn into a normal with the predicted spread $\sigma /\sqrt{n}$ and a skew that decays like $2/\sqrt{n}$.

Read the two columns: the standard deviation of the mean tracks $1/\sqrt{n}$ (0.447, 0.183, 0.071), and the skewness of the sampling distribution decays like the CLT’s $1/\sqrt{n}$ too (2.0 → 0.87 → 0.35 → 0.15). At $n=200$ the sampling distribution is essentially normal even though each single draw is maximally skewed. The running mean drifts around but converges to 1 - the LLN, not yet monotone at $n=100$.

---

4. Failure Modes & First-Principles Breakdowns

1. Reporting a point estimate with no error bar. $\bar{X}=0.08$ is meaningless without $\mathrm{SE}=s/\sqrt{n}$. “8% ± 2%” is a claim; “8%” is a guess. The single most common beginner error in quant reporting.
2. Reading the sampling distribution as the population. The distribution in the plot above is of the mean, not of daily returns. Daily returns stay skewed and fat-tailed forever; it is only the average that normalises. Confusing the two makes you under-state tail risk.
3. Forgetting that the CLT needs $n$ large and light tails. The Berry–Esseen bound depends on the third moment; a Pareto-tailed population (infinite variance) has no CLT at all in the $\sigma /\sqrt{n}$ form, and even finite-variance fat tails need large $n$.
4. The $1/\sqrt{n}$ trap. To double precision you need 4× the data; to get one more significant digit, 100×. Short backtests therefore carry irreducible estimation error no amount of cleverness removes.
5. Treating $1/\sqrt{\cdot }$ as $1/n$. Many people intuitively expect errors to fall proportionally to sample size. They fall like its square root - the reason estimation is hard.

---

5. References

• Casella & Berger, Statistical Inference
• Glasserman, Monte Carlo Methods in Financial Engineering
• Hastie, Tibshirani & Friedman, ESL
• Tsay, Analysis of Financial Time Series

---

6. Connected Graph Bridges

• Base: Probability & Measure Theory · Probability from Zero
• Continue: 02 · Point Estimation · 03 · The CLT & Sampling · Index Hub