1.5.1 Signal Processing & Kalman from Zero

Basic Prerequisites: none beyond high-school algebra - this page builds the idea from scratch.

---

1. Intuition & Practical Objective

Start with the dumbest question: what is a price, really? You see a number on a screen - say you watch a stock trade at 100.02, then 99.98, then 100.01, then 99.99. The fundamental value you actually care about is not jumping 4 cents every second. The tick-to-tick wiggles are bid–ask bounce, discreteness, and random order flow: a buyer and a seller disagreeing about a quarter of a cent, not a change in what the business is worth. That noise is real, it is large, and - this is the key point - it is not signal.

So the object of study is not the observed price $y_t$. It is the latent signal $x_t$ hiding inside it. We write the simplest possible model of this idea:

$$\underset{\text{observed price}}{\underbrace{y_t}}=\underset{\text{latent "true" value}}{\underbrace{x_t}}+\underset{\text{noise}}{\underbrace{e_t}}.$$

The practical objective of signal processing is to recover $x_t$ from a stream of noisy $y_t$‘s - optimally, in real time, with an honest accounting of how sure we are. The practical objective of the Kalman filter is to be that optimal recovery, and it delivers one thing no fixed-window method can: a running estimate plus its own uncertainty.

Three “aha”s carry the whole subject:

1. A price is not a number, it is a number plus an error bar. Every naive estimator (a raw price, a rolling average, a plain OLS slope) quotes a point estimate and pretends its error is zero. The right representation is a distribution: “the true beta is $1.02\pm 0.03$“. The filter carries both.
2. The trade-off is fundamental, not fixable by cleverness. Want to cut noise? Average more observations - but then you lag reality. Want to react instantly? Trust the latest tick - but then you inherit its noise. This is the bias–variance / lag–smoothness trade-off, and no filter escapes it. The Kalman filter’s contribution is not to break the trade-off but to let you choose the point on it explicitly (via $Q$ and $R$) and to do so optimally.
3. The filter is just Bayesian updating, recursively. Before the observation you hold a belief (“the value is $N(100.00,0.5^2)$”). The observation $y=100.02$ is evidence, itself noisy. The filter combines prior and evidence into a posterior - and, because the model is Gaussian, that combination is a weighted average whose weight is exactly the ratio of certainty. No new machinery, just Bayes applied over and over.

---

2. Mathematical Ground Truth & Derivations

The simplest signal model - a random walk. The latent value drifts as a random walk; we see it through noise:

$$x_{t+1}=x_t+w_t,w_t\sim N(0,q)\text{(signal: drifts by }\sqrt{q}\text{ each step)}$$ $$y_t=x_t+e_t,e_t\sim N(0,r)\text{(observation: corrupted by noise of size }\sqrt{r}).$$

Two numbers encode the whole problem: $q$ (how much the truth moves) and $r$ (how noisy each look is). Their ratio $q/r$ is the signal-to-noise ratio, and it decides everything - how much to trust the data versus your own prediction.

Deriving the filter from Bayes (the Gaussian-weight view). Suppose before seeing $y_t$ our belief about $x_t$ is Gaussian with mean $x^{-}$ and variance $P^{-}$ (the “a priori” - the prediction). The observation tells us $y_t=x_t+e_t$, i.e. it is $x_t$ plus independent noise of variance $r$. The posterior for $x_t$ is Gaussian with

$$\hat{x}=\underset{\text{prediction}}{\underbrace{x^{-}}}+K\underset{\text{innovation}}{\underbrace{(y_t-x^{-})}},P=(1-K)P^{-},K=\frac{P^{-}}{P^{-}+r}.$$

Read $K$ carefully: it is the fraction of the total variance that belongs to the signal. If the prediction is very uncertain ($P^{-}\gg r$), then $K\to 1$: trust the data, jump to it. If the data is very noisy ($r\gg P^{-}$), then $K\to 0$: ignore it, keep the prediction. $K$ is a shrinkage weight between two Gaussian beliefs, and it is the whole filter in one symbol.

The recursion - where prediction comes from. After updating at time $t$ we have $(x_{t\mid t},P_{t\mid t})$. The signal then drifts by one step of variance $q$, undoing none of our uncertainty:

$$P_{t+1\mid t}=P_{t\mid t}+q,x_{t+1\mid t}=x_{t\mid t}.$$

So the filter alternates forever: predict (add $q$ to uncertainty), update (shrink uncertainty by $K$ and move toward the data), repeat. That single line - “add process noise, then subtract information” - is the entire algorithm.

---

3. Computational Implementation - a filter that beats the raw price

A 200-line detour is unnecessary: the scalar filter above is 10 lines. The point of the demo is to measure how much noise the filter removes versus the raw observation, on data where we know the truth.

The filter cuts the error by 59% versus the raw price, with a steady-state gain of only $K^{\ast }=0.1318$ - because the signal moves so slowly ($q/r=0.02$) that the smart move is to treat each new tick as mostly noise and update only ~13% of the way toward it. This is the whole lesson: the optimal amount of the data to trust depends on $q/r$, and the filter computes it for you.

---

4. Failure Modes & First-Principles Breakdowns

1. “Just smooth it” is not a method. A moving average is a filter with a fixed, implicit gain and an even, useless-across-regimes weighting. Choosing its window by eyeball is choosing a point on the lag–noise curve arbitrarily. The Kalman filter makes the same choice derivable from $q,r$ - and updates it as uncertainty evolves.
2. Assuming the noise is white when it is not. Bid–ask bounce produces negatively autocorrelated price changes (Tsay Ch 5: $P_t=P_t^{\ast }+\frac{S}{2}{I}_t$ induces ${\rho }_1=-0.5$). A filter that treats consecutive observations as independent will over-count correlated noise as information. Model the noise structure, don’t just assume $e_t$ iid.
3. Confusing the latent state with the observable. The filter estimates $x_t$, the latent value; you still trade the observable $y_t$. If your fill price is $y_t$ but your edge is in $x_t$, the gap between them is a cost you must budget for.
4. The gain never fully settles if $q$ is wrong. $K^{\ast }=0.1318$ is optimal *for the assumed $q$. Feed the same filter a signal that actually moves 10× faster and $K^{\ast }$ is now far too small - the filter lags, and the “optimal” claim is void because it was optimal for the wrong model (see 05 · Failure Modes).

---

5. References

• Tsay, Analysis of Financial Time Series (3rd ed.)
• Welch & Bishop, An Introduction to the Kalman Filter (UNC TR 95-041)
• Kalman (1960), A New Approach to Linear Filtering and Prediction Problems
• Durbin & Koopman, Time Series Analysis by State Space Methods

---

6. Connected Graph Bridges

• Base: Probability & Measure Theory (Gaussian conditioning)
• Continue: 02 · State-Space Models · Index Hub
• Sibling: Stat-Arb & Pairs Trading (the spread is a signal-plus-noise problem)