7.8 Regime Classification: HMM & GMM

Basic Prerequisites: Statistics & Inference (maximum likelihood, Gaussian densities) and Probability & Measure Theory (Bayes’ rule, conditional expectation, Markov chains). The econometric twin of this folder lives in Regime Detection - that folder is the Hamilton-filter / Markov-switching / timeseries view; this folder is the machine-learning view: clustering (GMM / K-means), the EM algorithm, HMM as a probabilistic generative model, and regime-conditional supervised learning. (these are the folder-level prerequisites for pages 02–06; page 01 states its own, smaller, entry requirements)

1. Intuition & Practical Objective

Markets spend most of their time in a latent state you never observe directly - calm, low-volatility grind versus panic, spiking-volatility crash. If the data-generating process for returns changes regime, then a single unconditional mean/vol fit is wrong in both regimes, and a strategy tuned on calm data is fragile exactly when turbulence arrives. Regime classification is the ML answer to the same question the econometrician’s filter asks: learn the regimes from the data, and assign each observation (or each day) to the regime it came from.

Where this folder sits in the ML map:

1. Unsupervised regime detection. If a regime is a cluster in feature space (low vol, positive return vs high vol, negative return), then regime membership is a clustering problem - GMM / K-means on features 02.
2. The EM algorithm is the workhorse that fits every model in this folder: it is how GMMs are trained and how the HMM’s Baum–Welch step works 03.
3. HMM as a probabilistic regime model. Regimes are not independent draws - they persist and switch, so a hidden Markov chain over the latent state is the natural generative model; forward–backward + Viterbi give you the posterior state path 04.
4. Supervised regime labeling. When you have a regime proxy (a VIX threshold, an NBER recession flag, a drawdown rule), you can train a classifier on features instead of clustering - the bridge into regime-conditional supervised ML 06.

The one-sentence essence. “A regime is a latent label; the entire folder is two moves - (i) model the joint probability of observations and hidden regime (GMM: independent, HMM: Markov-persistent), and (ii) fit it by maximum likelihood via EM, then label each point by its posterior (responsibility, filtered, or Viterbi probability).“

2. Mathematical Ground Truth & Quick Lookup

Notation. $x_t\in {\mathbb{R}}^d$ observation/feature vector; $z_t\in \{1,\dots ,K\}$ latent regime; ${\pi }_k$ mixture weight / initial state prob; ${\mu }_k,{\Sigma }_k$ regime-conditional mean/covariance; $A_{ij}=\mathbb{P}(z_t=j\mid z_{t-1}=i)$ transition; ${\gamma }_t(k)=\mathbb{P}(z_t=k\mid \text{data})$ posterior; $N_k={\sum }_t{\gamma }_t(k)$ effective count.

GMM (independent mixture) - likelihood and EM (Dempster–Laird–Rubin 1977; ESL Ch 14.3):

HMM (Markov-persistent mixture) - forward–backward (Rabiner 1989; ESL Ch 14):

Critical caveat. The regime label is arbitrary. The likelihood is exactly invariant under a permutation of the states (label switching), so “state 0” carries no intrinsic meaning. Any downstream system that hard-codes “state 0 = calm” must instead impose an identifiability constraint (e.g. ${\sigma }_0<{\sigma }_1$, or ${\mu }_0>{\mu }_1$) at fit time - the exact degeneracy exploited in 05 · Failure Modes.

3. Computational Implementation - the whole toolbox in one file

Standard library + numpy only. The index gives the fastest demo of the folder’s central mechanism - EM drives the likelihood up monotonically (the guarantee behind both GMM and HMM training):

The full, working fits - K-means feature clustering, the GMM EM from scratch, the 2-D EM, the Baum–Welch HMM with Viterbi, and the failure-mode experiments - are the verified §3 blocks on the sub-pages, each reproduced byte-for-byte there.

4. Failure Modes & First-Principles Breakdowns (hub signposts)

The full analysis lives in 05 · Failure Modes & Practice. In one line each:

1. Label switching - the likelihood is symmetric under state permutation, so the same data can yield swapped labels (verified: two EM runs, LL both $2376.866$, labels reversed). Fix: an ordering constraint (${\sigma }_0<{\sigma }_1$) or a domain label.
2. Overfitting $k$ - adding a state always raises the likelihood; on genuine 2-regime data a 3-state fit splits a regime into near-clones (verified: LL $+1.35$ but BIC prefers $K=2$, $-4720.31$ vs $-4702.96$). Fix: penalize with BIC.
3. Look-ahead in regime labels - smoothed/decoded labels use future observations; using them at time $t$ is information leakage (verified: $+40$ pt agreement). Fix: label with the online filtered posterior, not the full-sample smoothed one.
4. Hard regime-switching is fragile to label error - with opposite-sign regime models, one wrong hard label flips the coefficient sign and can hurt (verified: hard-switch $-8.8\%$ vs soft-mixture $+5.5\%$).

5. References

• Hamilton, James D.: “A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle,” Econometrica 57(2):357–384, 1989
• Dempster, Arthur P., Laird, Nan M. & Rubin, Donald B.: “Maximum Likelihood from Incomplete Data via the EM Algorithm,” Journal of the Royal Statistical Society B 39(1):1–38, 1977
• Rabiner, Lawrence R.: “A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition,” Proceedings of the IEEE 77(2):257–286, 1989
• Ang, Andrew & Timmermann, Allan: “Regime Changes and Financial Markets,” Annual Review of Financial Economics 4:313–337, 2012
• Hastie, Tibshirani & Friedman, The Elements of Statistical Learning (2nd ed., 2009)
• Tsay, Ruey S., Analysis of Financial Time Series (3rd ed., 2010)

6. Connected Graph Bridges

• Foundational base: Statistics & Inference · Probability & Measure Theory · Linear Algebra & Matrices · Bayesian Statistics (EM as MAP/ML on latent variables) · Calculus & Optimization (the likelihood lower bound)
• Econometric sibling (do NOT duplicate - that folder is the filter/time-series view): Regime Detection · HMM (Hamilton filter view) · Regime-Detection Failure Modes
• ML siblings (this pillar): Financial ML Pitfalls & Low-SNR (the evaluation discipline every regime label inherits) · Tree & Boosting Methods (supervised labelers) · Purged & Embargoed CV (honest OOS for regime-conditional strategies) · ML for Portfolio Construction (regime-conditional allocation) · Deep Learning for Sequences
• Sub-pages (in-folder): 01 From Zero · 02 Unsupervised Clustering (GMM) · 03 The EM Algorithm · 04 HMM Regimes · 05 Failure Modes & Practice · 06 Advanced Extensions (Regime-Conditional ML)

Beginner: start at 01 · Practitioner: start at 05