1.5 Signal Processing & Kalman Filtering

Basic Prerequisites: Linear Algebra (matrix inverse, positive-definiteness) and Probability Theory (Gaussian conditioning, Bayes’ rule). (these are the folder-level prerequisites for pages 02–06; page 01 states its own, smaller, entry requirements)

1. Intuition & Practical Objective

A financial price series is almost never the object we care about. The observed price is a signal buried in noise: bid–ask bounce, discrete ticks, asynchronous trades, stale quotes, and transient order flow all add variance that carries no information about where the asset - or the relationship between two assets - is going. The quantity a quant actually wants (the latent fair value, the evolving hedge ratio, the conditional beta, the hidden volatility) is a state that is never observed directly and must be inferred.

The Kalman filter is the optimal recursive answer to that inference problem, under one honest assumption: the latent state and the observation are jointly linear-Gaussian. It threads a single, sharp needle. Static estimators (rolling OLS, fixed-window averages) force a crippling trade-off - long windows kill noise but lag every genuine regime change, short windows track change but are dominated by noise. A state-space model with a Kalman filter replaces that binary choice with an explicit, tunable balance between two covariances: how much the state is allowed to move per step (process noise $Q$) and how noisy each observation is (measurement noise $R$). The filter then does the optimum thing automatically - it trusts the data when the state is uncertain, trusts its own prediction when the data is noisy.

This folder is the topic-hub for signal processing & Kalman filtering in Kwant-Atlas. It (a) gives the fast recursion/formula lookup below - the job #1 of a hub - and (b) routes you to six sub-pages that walk from raw intuition through state-space models, the filter itself, time-varying beta, the failure modes, and the advanced extensions (smoothing, particle filters).

The one-sentence essence. “Every financial estimator is a filter; the Kalman filter is the provably optimal one - it prices each new observation by its signal-to-noise ratio and corrects the state by exactly the Kalman gain $K={\Sigma }^{-}{Z}^{\top }(Z{\Sigma }^{-}{Z}^{\top }+H{)}^{-1}$, the fraction of the innovation that is signal rather than noise.”

2. Mathematical Ground Truth & Derivations

Quick-Reference Lookup (job #1). Notation: $s_t$ latent state, $y_t$ observation, $s_{t\mid t-1}$ = a-priori (predicted) state, ${\Sigma }_{t\mid t-1}$ = a-priori state covariance, $v_t$ = innovation, $V_t$ = innovation covariance, $K_t$ = Kalman gain, $I$ = identity. All formulas follow Tsay Ch 11 Eq. (11.26)–(11.27) and (11.64) (verified in the corpus); the numbers in the check column were re-executed and reproduced exactly (see §3).

Linear-Gaussian state-space form (Tsay 11.26–11.27):

Critical convention caveat (Tsay 11.1). The filter comes in two equivalent forms and they must not be mixed within one pass. This hub uses the canonical (11.64) form, where ${\Sigma }_{t+1\mid t}$ is reached with $L_t=T_t-K_t{Z}_t$ and $K_t$ carries the $T_t$ factor (state one step ahead). The alternative contemporaneous form uses ${\Sigma }_{t+1\mid t}=T_t{\Sigma }_{t\mid t}{T}_t^{\top }+RQ{R}^{\top }$ and defines $K$ on the filtered step. Both give identical estimates for time-invariant models, but the state/covariance indices differ - mixing them silently off-by-ones every result.

3. Computational Implementation - the recursion engine

Standard-library only. It demonstrates the full predict → innovation → gain → update cycle on a hand-checkable one-step example, then verifies the steady-state (algebraic Riccati) gain.

The one-step output is exactly $x=5/6,P=5/9$ - the filter is doing the arithmetic $\hat{x}=0+\frac{1.25}{2.25}(1.5)$ and $P=\frac{1.25}{2.25}(1)$ by hand. The steady state $K_{\infty }=0.3904$ is the gain the filter settles to when it has seen enough data that its uncertainty no longer changes.

4. Failure Modes & First-Principles Breakdowns

Hub signposts - the folder’s failure-mode analysis lives in 05 · Failure Modes & Practice. In one line each:

1. Model misspecification ($Q,R$ wrong). The filter is optimal only for the model you hand it. Understate $Q$ and it lags every regime break; overstate $Q$ and it chases noise. Neither error is visible in the residuals unless you look.
2. Filter collapse & divergence. Set $Q\to 0$ (or let a badly conditioned covariance lose positive-definiteness) and the gain decays to zero: the filter stops listening to the data and diverges while reporting near-zero uncertainty - the signature is actual error $\gg$ reported standard error.
3. Linearity/Gaussianity. A Kalman filter is a linear minimum-variance estimator; it only equals the true posterior mean when the model is linear-Gaussian. Non-linear observation maps (stochastic volatility, duration, default) need the extended/unscented filter or a particle filter (page 06).
4. Lag is structural, not a bug. Any causal filter must trade tracking speed against smoothness; there is no filter that is both instantaneous and noise-free. The right question is how much lag a given $Q$ buys, not how to eliminate it.

5. References

• Tsay, Ruey S.: Analysis of Financial Time Series (3rd ed., 2010)
• Tsay, Ch 1–3
• Durbin, J. & Koopman, S. J.: Time Series Analysis by State Space Methods (2nd ed.)
• Harvey, A. C.: Forecasting, Structural Time Series Models and the Kalman Filter (1989)
• Kalman, R. E. (1960): A New Approach to Linear Filtering and Prediction Problems, J. Basic Engineering. The original paper.
• Welch, G. & Bishop, G.: An Introduction to the Kalman Filter, UNC-Chapel Hill TR 95-041
• Chan, Ernest P.: Algorithmic Trading: Winning Strategies and Their Rationale
• Särkkä, S.: Bayesian Filtering and Smoothing (2013)
• Hilpisch, Y. Python for Algorithmic Trading / Strimpel Python Algorithmic Trading Cookbook (both in the corpus)

6. Connected Graph Bridges

• Foundational base: Linear Algebra · Probability & Measure Theory · Econometrics & Time Series
• Sibling topics: Stat-Arb & Pairs Trading (dynamic hedge ratios) · Fundamental Multi-Factor Models (time-varying betas) · Signal Processing & Kalman Filtering (flat note)
• Cross-pillar: Regime Classification (HMM & GMM) (the hidden-state view beyond linear-Gaussian)
• Sub-pages (in-folder): 01 From Zero · 02 State-Space Models · 03 The Kalman Filter · 04 Time-Varying Beta · 05 Failure Modes · 06 Advanced Extensions

Beginner: start at 01 · Practitioner: start at 05