7.8.2 Unsupervised Regime Clustering

Basic Prerequisites: 01 · From Zero and Statistics & Inference (Gaussian density, maximum likelihood).

---

1. Intuition & Practical Objective

K-means (01) makes a hard call: every day belongs to exactly one cluster. But a regime call at $t$ is rarely certain - a single return near both regime means could come from either. The Gaussian Mixture Model (GMM) is the soft generalization: it models the data as a weighted sum of $K$ Gaussians, and each observation carries a responsibility ${\gamma }_t(k)\in [0,1]$ - the posterior probability it came from regime $k$. You keep the uncertainty, which is exactly what you want when the next step is “how much risk can I take given I might be wrong about the regime.”

The practical objective: fit the mixture’s parameters $({\pi }_k,{\mu }_k,{\Sigma }_k)$ so the model best explains the observed feature vectors, then use the responsibilities as soft regime membership. This is the workhorse unsupervised regime detector: fit it on (rolling-vol, rolling-return, drawdown, …) features, and each day gets a vector of regime probabilities instead of a hard flag.

The one-sentence essence. “Assume the data are a mixture of $K$ Gaussians (one per regime), then ask which Gaussian made this point - the responsibility is Bayes’ rule, and the parameters are fit so the whole mixture best predicts the data (maximum likelihood).“

---

2. Mathematical Ground Truth & Derivations

The model (a finite mixture; ESL Ch 14.3). The likelihood of the full sample, with $N$ observations $x_t\in {\mathbb{R}}^d$:

$$p(\mathbf{x}\mid \theta )=\prod _{t=1}^N\sum _{k=1}^K{\pi }_k\mathcal{N}(x_t;{\mu }_k,{\Sigma }_k),\theta =\{{\pi }_k,{\mu }_k,{\Sigma }_k{\}}_{k=1}^K,\sum _k{\pi }_k=1.$$

This has no closed-form maximum (the log of the sum does not separate), so we maximize it iteratively via EM (03). Define the responsibilities and effective counts:

$$N_k=\sum _{t=1}^N{\gamma }_t(k),{\gamma }_t(k)=\mathbb{P}(z_t=k\mid x_t)=\frac{{\pi }_k\mathcal{N}(x_t;{\mu }_k,{\Sigma }_k)}{{\sum }_j{\pi }_j\mathcal{N}(x_t;{\mu }_j,{\Sigma }_j)}.$$

E-step: fix $\theta$ and recompute ${\gamma }_t(k)$. M-step: with $\gamma$ fixed, the updates are weighted-averaging closed forms:

$${\pi }_k=\frac{N_k}{N},{\mu }_k=\frac{1}{N_k}\sum _t{\gamma }_t(k)x_t,{\Sigma }_k=\frac{1}{N_k}\sum _t{\gamma }_t(k)(x_t-{\mu }_k)(x_t-{\mu }_k{)}^{\top }.$$

In one dimension the covariance update reduces to ${\sigma }_k^2=\frac{1}{N_k}{\sum }_t{\gamma }_t(k)(x_t-{\mu }_k{)}^2$. Each M-step increases the log-likelihood; EM converges to a local maximum of the mixture likelihood (Dempster–Laird–Rubin 1977; the monotonicity proof is 03).

Soft vs hard. K-means is the $\gamma \to \{0,1\}$ limit (hard nearest-mean assignment, equal weights, diagonal covariance). GMM keeps fractional responsibilities - so a day that sits between two regime means can carry, say, $\gamma =(0.4,0.6)$ rather than a forced $0/1$. That fractional membership is what regime-conditional risk sizing needs (06).

---

3. Computational Implementation - fit a GMM by EM from scratch

Fit a 2-regime GMM to simulated daily returns (calm low-vol bull, stress high-vol) by EM from scratch, and read off the recovered regime means/vols and the log-likelihood’s monotone climb. Stdlib only.

The EM run recovers both regime volatilities to within a few percent of the truth ($0.00794$ vs $0.008$ - 0.75% low; $0.02686$ vs $0.028$ - 4.1% high) and the weights ($0.780$ calm / $0.220$ stress) match the simulated regime occupancy - the mixture has learned the two regimes from unlabeled returns. The log-likelihood climbs $1740.73\to 2968.43$ monotonically and flatlines (last-20 change $0$) at convergence - the EM guarantee, verified in numbers. The responsibilities ${\gamma }_t(k)$ are your soft regime labels, ready to feed a regime-conditional model (06).

---

4. Failure Modes & First-Principles Breakdowns

1. $K$ is not known - more Gaussians always fits better; the mixture will happily split one regime into clones (verified overfit in 05). Fix: BIC or gap statistic (ESL §14.3.11).
2. Label switching - the mixture likelihood is symmetric under state permutation; two EM runs can return swapped states with identical likelihood (verified: both $2376.866$). Fix: order-constrain ${\sigma }_0<{\sigma }_1$.
3. EM is local - it converges to a local maximum; a bad initialization lands on a worse fit. Fix: many random restarts, keep the highest-likelihood one.
4. Independence is wrong for time series - a GMM treats days as independent draws, ignoring regime persistence. The HMM (04) adds the Markov chain over states precisely to fix this.

---

5. References

• Hastie, Tibshirani & Friedman, The Elements of Statistical Learning
• Dempster, Laird & Rubin, “Maximum Likelihood from Incomplete Data via the EM Algorithm,” JRSS-B 39(1), 1977
• Tsay, Analysis of Financial Time Series

---

6. Connected Graph Bridges

• Back: 01 · From Zero
• Forward: 03 · The EM Algorithm · 04 · HMM Regimes
• Base: Statistics & Inference · Probability & Measure Theory