1.8 Regime Detection

Basic Prerequisites: Econometrics & Time Series (stationarity, ARMA, forecasting) and Probability & Measure Theory (conditional expectation, Markov chains, Bayes’ rule). (these are the folder-level prerequisites for pages 02–06; page 01 states its own, smaller, entry requirements)

1. Intuition & Practical Objective

Markets do not sit in one statistical state. Sometimes they grind upward with calm, low-volatility returns (bull); sometimes they fall hard with spiking volatility (bear); sometimes vol rises without direction (regime). If the data-generating process for returns changes regime, then a single unconditional mean/vol fit is wrong for both regimes, and - worse - a strategy tuned on calm data is fragile exactly when turbulence arrives. Regime detection is the discipline of learning the regime from the data and knowing which regime you are in right now.

This folder is the topic-hub for regime detection in Kwant-Atlas. It (a) gives the fast formula lookup below - job #1 of a hub - and (b) routes to six sub-pages that walk from raw intuition, through Markov-switching models (Hamilton), threshold models (SETAR/STAR), hidden Markov models (HMM), the failure modes (label switching, overfitting regimes, persistence), and regime-aware allocation.

The one-sentence essence. “Regimes are latent - you never observe bull/bear directly, only noisy returns - so the entire machinery is a recursive Bayes exercise: maintain a probability distribution over the hidden regime, update it with each new return, and estimate the regime parameters (means, vols, transition probabilities) by maximum likelihood via the Hamilton filter (or EM for the HMM).“

2. Mathematical Ground Truth & Derivations

Quick-Reference Lookup (job #1). All formulas below are transcribed from Hamilton (1989), Tsay Ch 4 (Markov switching / SETAR / STAR) and Tsay Ch 11–12 (state-space / MCMC), and cross-checked against the verified corpus. Numbers in the check column were re-executed and reproduced exactly from the working Python in §3 and the sub-pages.

Notation: $s_t\in \{0,1\}$ latent regime; $P_{ij}=\mathbb{P}[s_t=j\mid s_{t-1}=i]$ transition probability (rows $i$, columns $j$); ${\mu }_j,{\sigma }_j$ regime-conditional mean/vol of returns $y_t$; ${\hat{\xi }}_{t\mid t}=\mathbb{P}[s_t\mid y_1,\dots ,y_t]$ the filtered state distribution; $n(x)$ the standard-normal density.

Critical caveat. The regime label is arbitrary: the decision of which state to call “bull” vs “bear” is a normalization (Hamilton 1989 §4.2 fixes it by, e.g., ${\mu }_1>{\mu }_0$). Without that constraint the likelihood is exactly symmetric - which is the label-switching degeneracy exploited in 05 · Failure Modes.

3. Computational Implementation - the formula engine

Standard library only. Reproduces the two cleanest verified numbers from §2: Hamilton’s expected regime durations and the normalized two-state filter (probabilities stay in $[0,1]$ and sum to $1$ by construction - Hamilton’s §4.2 point vs the Liptser–Shiryayev continuous-time analog).

Starting from the correct stationary prior ($P(\text{expansion})=0.72$), the filter stays expansion-leaning and firms up slightly as the two positive observations arrive - the qualitative essence of regime detection, reproduced on three data points. (Note how sensitive the path is to the prior: mislabel the states and the same data appear to “start in recession”, which is exactly the identification problem the caveat below warns about.)

4. Failure Modes & First-Principles Breakdowns

Hub signposts - the folder’s full failure-mode analysis lives in 05 · Failure Modes & Practice. In one line each:

1. Label switching. The likelihood is symmetric under a permutation of the states, so two identical models can be returned with swapped labels (verified: two EM runs, loglik both $1309.234$, means permuted). Fix by a parameter constraint (${\mu }_1>{\mu }_0$).
2. Overfitting regimes. Adding a state always raises the likelihood; a 3-state HMM on genuine 2-state data splits one regime into two near-identical clones (verified: loglik $+0.925$ but BIC prefers 2-state, $-2580$ vs $-2543$).
3. Regime persistence & late detection. The filter is Bayesian and slow by design; on volatile-but-meaningless data it oscillates, and every “regime” call carries detection lag that a fast strategy pays for.

5. References

• Hamilton, James D.: A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle, Econometrica 57(2), 357–384 (1989). *Verified corpus ref (pillar1 PDF, pdftotext deep-read). The foundational paper: the nonlinear filter, ML estimation, and the 1952–1984 US GNP application
• Tsay, Ruey S.: Analysis of Financial Time Series, 3rd ed. (2010)
• Ang, Andrew & Timmermann, Allan: Regime Changes and Financial Markets, Annual Review of Financial Economics 4, 313–337 (2012). *The canonical survey linking estimated regimes to fat tails, heteroskedasticity, skewness, and portfolio choice
• Kritzman, Mark, Page, Sébastien & Turkington, David: Regime Shifts: Implications for Dynamic Strategies, Financial Analysts Journal 68(3) (2012). Practitioner application of regime detection (Markov-switching on macro data) to dynamic asset allocation.

6. Connected Graph Bridges

• Foundational base: Econometrics & Time Series · Probability & Measure Theory · Statistics & Inference · Bayesian Statistics
• Sibling topics (this pillar): Signal Processing & the Kalman Filter (the continuous-state sibling of the discrete-state filter here) · Momentum (trend = a persistence regime) · Statistical Arbitrage & Pairs (mean-reversion = a different regime)
• ML cross-link: Regime Classification: HMM & GMM (Baum–Welch, Viterbi, regime-aware ML)
• Risk & portfolio: ES) · Stress Testing & Scenario Analysis · Risk Parity · Black–Litterman
• Sub-pages (in-folder): 01 From Zero · 02 Markov-Switching Models · 03 Threshold Models · 04 Hidden Markov Models · 05 Failure Modes & Practice · 06 Advanced Extensions

Beginner: start at 01 · Practitioner: start at 05