M.7.2 Bayes' Theorem, Priors & Conjugate Families

Basic Prerequisites: 01 · From Zero and Probability & Measure Theory.

---

1. Intuition & Practical Objective

Page 01 gave the update rule. This page builds the dictionary that makes it usable: which prior pairs with which likelihood so the posterior stays in a known family (conjugacy), what it means to choose a noninformative or improper prior, and where the marginal likelihood (the evidence) earns its keep for model comparison.

The practical objective is a lookup you can carry: given my likelihood, what is the conjugate prior, what is the posterior, and what are its parameters? Get this right and every “update” becomes arithmetic; get it wrong (an improper prior whose posterior is also improper) and your whole inference is silently undefined.

The one-sentence essence. “A conjugate prior is a prior from the same exponential family as the posterior, so updating is bookkeeping - and the marginal likelihood $m(x)$ is the same integral whether you use it to normalize or to compare models.”

---

2. Mathematical Ground Truth & Derivations

Bayes’ theorem, twice. The event form is $P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)}$; the parameter form is C&B eq. 7.2.6:

$$p(\theta \mid x)=\frac{f(x\mid \theta )\pi (\theta )}{m(x)},m(x)={\int }_{\Theta }f(x\mid \theta )\pi (\theta )d\theta .$$

The numerator is the joint density $f(x,\theta )$; the denominator is its integral over $\theta$ - the probability of the data under the prior. Every Bayesian quantity is a functional of the numerator after normalization.

Conjugacy (C&B Def 7.2.2). A class $\Pi$ of priors is conjugate for a likelihood class $\mathcal{F}$ if the posterior stays in $\Pi$ for every $f\in \mathcal{F}$, every prior in $\Pi$, and every $x$. The exponential families generate these pairs (the prior’s exponent is chosen to complete the likelihood’s exponent). The working dictionary:

Likelihood	Conjugate prior	Posterior update	Posterior mean
$\mathrm{Bernoulli}(p)$ / $\mathrm{Bin}(n,p)$	$\mathrm{Beta}(a,b)$	$\mathrm{Beta}(a+y,b+n-y)$	$\frac{a+y}{a+b+n}$
$\mathrm{Poisson}(\lambda )$	$\mathrm{Gamma}(\alpha ,\beta )$	$\mathrm{Gamma}(\alpha +\sum y_i,\beta +n)$	$\frac{\alpha +\sum y_i}{\beta +n}$
$N(\theta ,{\sigma }^2)$, ${\sigma }^2$ known	$N({\mu }_0,{\tau }^2)$	precision adds (below)	${\tau }_{\text{post}}^2(\frac{{\mu }_0}{{\tau }^2}+\frac{n\bar{x}}{{\sigma }^2})$
$N(\mu ,{\sigma }^2)$, $\mu$ known	$\mathrm{InvGamma}(\alpha ,\beta )$	$\mathrm{InvGamma}(\alpha +\frac{n}{2},\beta +\frac{1}{2}\sum (y_i-\mu {)}^2)$	$\frac{\beta +\frac{1}{2}\sum (y_i-\mu {)}^2}{\alpha +\frac{n}{2}-1}$
$\mathrm{Multinomial}(n,\mathbf{p})$	$\mathrm{Dirichlet}(\mathbf{\alpha })$	$\mathrm{Dirichlet}(\mathbf{\alpha }+\mathbf{y})$	$\frac{{\alpha }_j+y_j}{{\sum }_k({\alpha }_k+y_k)}$
$N(\mathbf{\theta },{\sigma }^{2}I)$, ${\sigma }^2$ known	$N({\mathbf{\mu }}_0,{\Sigma }_0)$	${\Sigma }_{\text{post}}^{-1}={\Sigma }_0^{-1}+{\sigma }^{-2}{X}^{\top }X$	${\Sigma }_{\text{post}}({\Sigma }_0^{-1}{\mathbf{\mu }}_0+{\sigma }^{-2}{X}^{\top }\mathbf{y})$

Normal–Normal with known variance (C&B Ex 7.2.10) - the precision result. With $X_i\sim N(\theta ,{\sigma }^2)$ and $\theta \sim N({\mu }_0,{\tau }^2)$,

$$\frac{1}{{\tau }_{\text{post}}^2}=\frac{1}{{\tau }^2}+\frac{n}{{\sigma }^2},{\mu }_{\text{post}}={\tau }_{\text{post}}^2(\frac{{\mu }_0}{{\tau }^2}+\frac{n\bar{x}}{{\sigma }^2}).$$

Precisions add. Information (reciprocal variance) is additive: the posterior precision is the prior precision plus the data precision $n/{\sigma }^2$. This is the single most useful Bayesian intuition - and its matrix version (${\Sigma }_{\text{post}}^{-1}={\Sigma }_0^{-1}+{\sigma }^{-2}{X}^{\top }X$) is the same statement for vectors. As ${\tau }^2\to \infty$ (vague prior) ${\mu }_{\text{post}}\to \bar{x}$: the datum wins when the prior has no precision.

The multivariate normal prior is the Gaussian-process/regression prior. With a design matrix $X$, the posterior above is a posterior over regression coefficients, and its mean is the ridge solution (§04) - the deep link between Bayesian regression and penalized least squares. See Linear Algebra for precision matrices and Gaussian conditionals.

Sequential vs batch. Because $m(x)={\prod }_i{m}_i$ factorizes for independent data, updating observation-by-observation (using the current posterior as the next prior) equals the batch update. The order of the data is irrelevant - a property frequentists’ estimators also share, but which Bayes makes visible.

Noninformative and improper priors. Laplace’s $\pi (\theta )\propto 1$ (“flat”) or Jeffreys’ $\pi (\theta )\propto \sqrt{\det I(\theta )}$ (invariant under reparameterization) are attempts to “let the data speak.” They are typically improper - they do not integrate to 1 - and are tolerated only because the resulting posterior is proper ($m(x)<\infty$). Flat priors over unbounded parameters are the standard example; the “improper prior” §04 page warns when this breaks.

Marginal likelihood / evidence and Bayes factors. For two models ${\mathcal{M}}_0,{\mathcal{M}}_1$,

$${\mathrm{BF}}_{10}=\frac{m_0(x)}{m_1(x)}=\frac{\int f(x\mid {\theta }_0){\pi }_0({\theta }_0)d{\theta }_0}{\int f(x\mid {\theta }_1){\pi }_1({\theta }_1)d{\theta }_1},$$

and $m(x)$ is exactly the denominator Bayes’ rule dropped for estimation. It is the automatic Occam’s razor: a model that spreads prior mass over implausible regions pays a smaller $m(x)$. BIC, $\mathrm{BIC}=-2\log \hat{L}+(\log n)d$ (ESL eq. 7.35), is a large-sample $-2\log m(x)$ approximation - the bridge to frequentist model selection.

---

3. Computational Implementation - the four conjugate updates, verified

Run the four canonical conjugate updates, confirm that sequential updating equals batch updating (Normal–Normal), and check the “precision adds” law. Stdlib only.

Three facts are visible in the output. (i) Precisions add: $10.250=0.250+10.000$, so the data precision $n/{\sigma }^2=10$ dwarfs the prior’s $0.25$ and the posterior mean $1.2683$ sits close to $\bar{x}=1.30$. (ii) Sequential = batch, for both Beta–Bernoulli and Normal–Normal - the order of the data does not matter. (iii) The Gamma–Poisson posterior mean $2.3636$ sits between the prior mean $2.0$ and the sample mean $2.4$, exactly as shrinkage requires.

---

4. Failure Modes & First-Principles Breakdowns

1. Improper prior, improper posterior (the silent killer). $\pi (\theta )\propto 1$ over an unbounded parameter does not integrate to 1. The posterior is only legitimate if $m(x)=\int f(x\mid \theta )\pi (\theta )d\theta <\infty$. With, e.g., a $1/\theta$ flat prior on a variance that can concentrate mass at 0, or with improper hierarchical hyper-priors, the integral diverges and every “estimate” is meaningless. Always check that the posterior is proper (or use a proper, weakly-informative prior - BDA3’s default recommendation).
2. Prior–data conflict hidden by conjugacy. Conjugacy gives a closed form whether or not the prior is sane. If the prior says $\theta \approx 0$ and the data scream $\theta \approx 5$ with high precision, the posterior will be a battle of precisions - and with a strong prior, wrong. Plot prior and likelihood before trusting the posterior.
3. Choosing a conjugate prior for convenience, then forgetting it is a model. A Normal prior is not “natural” for a strictly positive quantity; a $\mathrm{Gamma}$ is. Conjugate-ness is a tool, not a justification (C&B: “whether a conjugate family is reasonable… is a question to be left to the experimenter”).
4. Dropping $m(x)$, then asking for a Bayes factor. You may ignore the evidence to estimate $\theta$; you may not ignore it to compare models. Reusing an unnormalized posterior across models and comparing heights is a common and fatal error.
5. Flat priors are not “objective.” A flat prior on $\theta$ is non-flat on ${\theta }^2$, $\log \theta$, etc. Jeffreys’ prior fixes the most obvious reparameterization invariance, but “noninformative” is a claim to defend, not assume.

---

5. References

• Casella & Berger, Statistical Inference (2nd ed.)
• Gelman et al., Bayesian Data Analysis (3rd ed.)
• Tsay, Analysis of Financial Time Series
• Hoff, A First Course in Bayesian Statistical Methods
• MacKay, Information Theory, Inference, and Learning Algorithms

---

6. Connected Graph Bridges

• Back: 01 · From Zero · Index Hub
• Continue: 03 · Posterior Inference (what to do with the posterior) · 04 · Bayesian & Regularization (the Gaussian prior is ridge)
• Base: Probability & Measure Theory (conditional distributions, densities) · Linear Algebra (precision matrices, Gaussian conditionals)
• Forward: Black–Litterman Allocation (a normal–normal update on expected returns)