M.7.1 Bayesian Statistics from Zero

Basic Prerequisites: Probability & Measure Theory (conditional probability, densities).

---

1. Intuition & Practical Objective

This page builds the why of Bayesian statistics with no prior statistics beyond conditional probability needed. The objective is one idea: learning is Bayes’ rule. You start with a belief about the world (a distribution over the unknown), you see data, and you multiply the belief by how likely that data was, then renormalize. The result - the posterior - is your complete, updated state of knowledge.

Start with the dumbest question: what is a probability, really? A frequentist says: the long-run relative frequency of an event in repeated trials. A Bayesian says: a degree of belief, coherent enough to bet on. For a fair coin the two agree. For “will this fund be above its benchmark next year?” or “what is the mean return of this strategy?” the frequentist has nothing to assign a probability to - there is one realization, one history, one parameter. The Bayesian does: probability is a state of knowledge, and it updates by a fixed rule.

Three “aha”s:

1. Bayes’ theorem is just conditional probability, read backwards. $P(A\mid B)$ and $P(B\mid A)$ are not the same thing (this is the classic base-rate fallacy), and Bayes’ rule is the exact conversion between them. The update below is nothing more than $P(\theta \mid x)=P(x\mid \theta )P(\theta )/P(x)$ applied to a parameter $\theta$ instead of an event.

2. The posterior is the whole answer, not a number. $\bar{x}$ (the MLE) is one summary; the posterior $p(\theta \mid x)$ is a full distribution over plausible values, and every estimate, interval, and prediction is a functional of it. You keep the uncertainty instead of throwing it away at the first step.

3. Conjugate priors make the update a one-line arithmetic. If the prior and likelihood come from matching families, you never integrate anything: the posterior is the same family with updated parameters. Beta prior + Bernoulli data $\Rightarrow$ Beta posterior; the update is literally “add the successes to $a$, add the failures to $b$.” This is why practical Bayesians live in conjugate families - and why Gibbs sampling (§05) works.

---

2. Mathematical Ground Truth & Derivations

Bayes’ rule for a parameter (C&B eq. 7.2.6–7.2.7). Let $\theta$ be the unknown with prior density $\pi (\theta )$ (C&B: the classical parameter is now random, with a subjective distribution fixed before the data). Given data $x$ with likelihood $f(x\mid \theta )$, the posterior is

$$p(\theta \mid x)=\frac{f(x\mid \theta )\pi (\theta )}{m(x)},m(x)=\int f(x\mid \theta )\pi (\theta )d\theta .$$

The denominator $m(x)$ - the marginal likelihood or evidence - does not depend on $\theta$, so the whole content of the update is the proportionality

$$\boxed{p(\theta \mid x)\propto f(x\mid \theta )\pi (\theta )}$$

posterior $\propto$ likelihood $\times$ prior. Renormalization is the only step that needs an integral, and for conjugate pairs even that is done for us by the known normalizing constant of the family.

The Beta–Bernoulli derivation, from scratch. Let $X_1,\dots ,X_n$ be iid $\mathrm{Bernoulli}(p)$ with $y=\sum X_i$ successes, and give $p$ a $\mathrm{Beta}(a,b)$ prior:

$$\pi (p)=\frac{1}{B(a,b)}{p}^{a-1}(1-p{)}^{b-1},B(a,b)=\frac{\Gamma (a)\Gamma (b)}{\Gamma (a+b)}.$$

The likelihood is $\left(\genfrac{}{}{0ex}{}{n}{y}\right)p^y(1-p{)}^{n-y}$, so

$$p(p\mid y)\propto \underset{\text{likelihood}}{\underbrace{p^y(1-p{)}^{n-y}}}\cdot \underset{\text{prior}}{\underbrace{p^{a-1}(1-p{)}^{b-1}}}=p^{a+y-1}(1-p{)}^{b+n-y-1},$$

which is exactly the kernel of $\mathrm{Beta}(a+y,b+n-y)$ (C&B Ex 7.2.9). So the conjugate update is:

$$\mathrm{Beta}(a,b)+(y\text{ successes in }n)⟶\mathrm{Beta}(a+y,b+n-y).$$

The posterior mean is a precision-weighted blend of prior mean and sample mean (C&B):

$$\mathbb{E}[p\mid y]=\frac{a+y}{a+b+n}=\underset{\text{weight on data}}{\underbrace{\frac{n}{a+b+n}}}\cdot \frac{y}{n}+\underset{\text{weight on prior}}{\underbrace{\frac{a+b}{a+b+n}}}\cdot \frac{a}{a+b}.$$

This single formula is shrinkage: the MLE $y/n$ is pulled toward the prior mean $a/(a+b)$ by an amount that vanishes as $n\to \infty$. With a uniform $\mathrm{Beta}(1,1)$ prior the posterior mean is $(y+1)/(n+2)$ - Laplace’s rule of succession, the answer to “what happens if you’ve seen 3 heads in 3 tosses?” (not $p=1$, but $p=4/5$).

Sequential updating. Bayes’ rule is idempotent under batching: updating one observation at a time, reusing the previous posterior as the new prior, gives the same posterior as multiplying all the likelihoods at once. That is why the update above is “add the counts” - order never matters.

---

3. Computational Implementation - watch a belief update

Start with a mild prior $\mathrm{Beta}(2,2)$ (belief the coin is roughly fair), observe 8 flips (5 heads), and see the posterior move. Then check the whole update by brute-force grid integration - the analytic posterior and the numerically integrated one must agree. Stdlib only.

Read the three lines at the bottom together: the same 8 flips produce a posterior mean of $0.60$ under a uniform prior, $0.5833$ under the mild prior, and $0.5208$ under the informative $\mathrm{Beta}(20,20)$ prior - which has the weight of 40 pseudo-observations, so eight real ones barely nudge it. The grid check ($0.5833$ vs $0.5833$) confirms the analytic posterior is exactly right.

---

4. Failure Modes & First-Principles Breakdowns

1. Prior sensitivity (the sample is small, the prior is loud). The last line of the run shows a $\mathrm{Beta}(20,20)$ prior swallowing 8 observations. With $n$ data points and a prior worth $a+b$ pseudo-counts, the data weight is $n/(a+b+n)$ - you must know how much prior you injected. Never “hide” prior information in the prior; report it and test sensitivity.
2. Base-rate neglect - doing the ratio backwards. Confusing $P(\text{disease}\mid +)$ with $P(+\mid \text{disease})$ is the single most common probabilistic error, and it is the same error as confusing $p(\theta \mid x)$ with $f(x\mid \theta )$. Low base rates (a rare edge, a rare regime) make the reversal catastrophic in magnitude.
3. Treating the prior as “a belief” when it is really a regularizer. In high dimensions the prior is a modeling device that makes the estimate well-posed (ridge/lasso, §04), not a literal belief. Justify it as a stabilizer, and tune it honestly (cross-validation, empirical Bayes) rather than pretending it came from introspection.
4. Forgetting the evidence $m(x)$ is not optional for model comparison. For estimating $\theta$ you may drop $m(x)$; for comparing models you may not - $m(x)$ is the Bayes factor denominator. See 02 · Bayes’ Theorem & Priors.

---

5. References

• Casella & Berger, Statistical Inference (2nd ed.)
• Gelman et al., Bayesian Data Analysis (3rd ed.)
• McElreath, Statistical Rethinking (2nd ed.)
• Tsay, Analysis of Financial Time Series

---

6. Connected Graph Bridges

• Base: Probability & Measure Theory (conditional probability, Bayes’ rule)
• Continue: 02 · Bayes’ Theorem & Priors · 03 · Posterior Inference · Index Hub
• Forward (applications): Black–Litterman Allocation · Covariance Shrinkage