M.7 Bayesian Statistics

Basic Prerequisites: Probability & Measure Theory and Econometrics & Time Series. (these are the folder-level prerequisites for pages 02–06; page 01 states its own, smaller, entry requirements)

1. Intuition & Practical Objective

Classical (frequentist) statistics treats the parameter $\theta$ as an unknown fixed number and reasons about the data it would generate. Bayesian statistics treats $\theta$ as a random quantity with a distribution - the prior - and learns by multiplying the prior by the likelihood and renormalizing. The product is the posterior $p(\theta \mid x)$, and it is the entire answer: every Bayesian point estimate, interval, and prediction is a functional of it (Casella & Berger §7.2.3).

This matters in finance for one blunt reason: with short samples and many parameters, you cannot afford to ignore prior information. Portfolio weights are famously “estimation-error maximizers” (see MPT & Mean–Variance); Black–Litterman is Bayesian blending; covariance shrinkage is a Bayesian estimator; regime models are hidden-state posteriors. Bayesian statistics is the native language for all of it.

This folder is the topic-folder for that toolbox. This page is the hub: it (a) gives the fast formula lookup below (job #1), and (b) routes you to six sub-pages from raw intuition through Bayes’ theorem and priors, posterior inference, the regularization correspondence, MCMC, and hierarchical models.

The one-sentence essence. “Learning is Bayes’ rule: posterior $\propto$ likelihood $\times$ prior; conjugate priors make the update closed-form; the posterior is the answer; and when it has no closed form you sample it with a Markov chain (MCMC).”

The four load-bearing objects.

1. Prior $\pi (\theta )$ - what you believed before the data. In practice, a regularization device as much as a belief (§02, §04).
2. Likelihood $f(x\mid \theta )$ - the data-generating model (this is the same object a frequentist uses).
3. Posterior $p(\theta \mid x)$ - the complete, updated state of knowledge.
4. Posterior predictive $p(\tilde{y}\mid x)$ - the distribution of a new observation after integrating out $\theta$; the object you actually trade on (§03).

2. Mathematical Ground Truth & Formula Lookup

Quick-Reference Lookup (job #1). Definitions and conjugate results transcribed from Casella & Berger Ch 7 (§7.2.3 Bayes estimators, Def 7.2.2 conjugacy) and Tsay Ch 12 (§12.3 Bayesian inference; §12.4 Metropolis–Hastings; §12.8 FFBS), cross-checked against Gelman et al. (BDA3) and ESL Ch 3/§8.3. The “verified check” column was re-executed and reproduced exactly by the scripts in §3 and on the sub-pages.

Notation: $x=(x_1,\dots ,x_n)$ data; $\theta$ parameter; $\pi (\theta )$ prior; $f(x\mid \theta )$ likelihood; $p(\theta \mid x)$ posterior; $m(x)=\int f(x\mid \theta )\pi (\theta )d\theta$ marginal likelihood/evidence; $a,b$ Beta shape params; ${\tau }^2,{\sigma }^2$ between-/within-group variances; ${\sigma }^2$ is the noise variance, ${\tau }^2$ the signal/heterogeneity variance (hierarchical).

Conjugacy dictionary (the practical engine of Gibbs). Normal prior on a normal mean; Gamma on a Poisson rate; Beta on a Bernoulli $p$; Dirichlet on multinomial probabilities; Inverse-Gamma on a normal variance; Normal–Inverse-Gamma jointly on $(\mu ,{\sigma }^2)$. If the prior is conjugate, the full conditional is often a known family - and a Gibbs step is a closed-form draw.

3. Computational Implementation - one stdlib sanity sweep

Reproduces the check column of every closed-form row above in a single standard-library run.

The full runnable demonstrations (with output) live on the sub-pages: conjugate updating (§01–02), posterior inference and credible intervals (§03), ridge/lasso as MAP (§04), Metropolis–Hastings and Gibbs from scratch (§05), and hierarchical shrinkage (§06).

4. Failure Modes & First-Principles Breakdowns

Hub signposts - the folder’s failure analysis lives on the sub-pages. In one line each:

1. Prior sensitivity. With little data the posterior is the prior; an informative prior can dominate 8 observations entirely (Beta(20,20) barely moves on 5/8, §01). The prior is a modeling choice with consequences, not a formality.
2. Improper priors with improper posteriors. A prior $\pi (\theta )\propto 1$ does not integrate to 1; the “posterior” is only well-defined if $m(x)<\infty$. Improper prior does not guarantee proper posterior - a silent, catastrophic bug (§02).
3. Credible $\ne$ confidence. A 95% Bayesian credible interval says $\mathbb{P}(\theta \in C\mid x)=0.95$; a 95% frequentist confidence interval says nothing of the kind about a fixed $\theta$. Quoting one as the other is a category error (§03).
4. MCMC that has not converged. A sampler reports numbers whether or not the chain explored the posterior. Poor mixing under-converges to a mode and biases the estimate (the scale-$0.5$ run reports $\bar{\theta }=-0.72$ for a target mean of $-0.50$, §05). Always diagnose ($\hat{R}$, trace, acceptance rate).
5. Over-shrinkage under a hierarchical model. As the between-group variance ${\tau }^2\to 0$, every group mean is pulled to the grand mean and genuine heterogeneity is erased; with few groups ${\tau }^2$ is itself poorly identified (§06).

5. References

• Gelman, Carlin, Stern, Dunson, Vehtari & Rubin: Bayesian Data Analysis (3rd ed., 2013)
• McElreath, Richard: Statistical Rethinking (2nd ed., 2020)
• Hoff, Peter D.: A First Course in Bayesian Statistical Methods (2009)
• Robert, C. P. & Casella, G.: Monte Carlo Statistical Methods (2nd ed., 2004)
• Tsay, Ruey S.: Analysis of Financial Time Series (3rd ed., 2010)
• Casella, G. & Berger, R. L.: Statistical Inference (2nd ed., 2002)
• Hastie, Tibshirani & Friedman: The Elements of Statistical Learning (2nd ed., 2009)
• MacKay, D. J. C.: Information Theory, Inference, and Learning Algorithms (2003)
• Berger, J. O.: Statistical Decision Theory and Bayesian Analysis (2nd ed., 1985) and Robert, C. P.: The Bayesian Choice (2nd ed., 2001)

6. Connected Graph Bridges

• Foundational base: Probability & Measure Theory (Bayes’ rule, conditional expectation) · Linear Algebra (precision matrices, Gaussian conditionals) · Multivariable Calculus (MAP = optimization, convexity)
• Sibling foundations: Econometrics & Time Series (MLE, state-space/Kalman - the frequency-side twin of §05–06) · Numerical Methods (Monte Carlo; MCMC is “MC with dependence”) · Stochastic Calculus (measure change; the Radon–Nikodym view of Bayes)
• Sub-pages (in-folder): 01 From Zero · 02 Bayes’ Theorem & Priors · 03 Posterior Inference · 04 Bayesian & Regularization · 05 MCMC · 06 Advanced Extensions

Beginner: start at 01 · Practitioner: start at 05