7.6.1 Deep Learning for Sequences from Zero

Basic Prerequisites: Financial ML Pitfalls & Low SNR. Everything else is built from scratch on this page.

---

1. Intuition & Practical Objective

This page builds the why of sequence models with no prior deep-learning knowledge needed. The objective is one idea: if the order of your data carries information, then any model that ignores the order - a regression, a tree, a plain feed-forward net - is provably throwing that information away.

Start with the dumbest possible question. You have a stream $x_1,x_2,\dots ,x_T$ and you want to predict $y_t$. What does a normal (IID) model do? It treats each row $(x_t,y_t)$ as an independent draw and learns a function $f(x_t)\to y_t$. It is blind to the arrow of time: shuffle your rows, retrain, and the IID model is unchanged. If shuffling changes nothing, the model has discarded everything the ordering could have told it.

Financial series are ordered in exactly the way that matters. Three “aha”s:

1. There is state, and it is not in the current row. Whether the market is in a trending or mean-reverting mood is not visible in today’s return alone - it is visible in the sequence of recent returns. A model with no memory cannot represent “we have been drifting up for two weeks.”
2. Order is information, and shuffling destroys it. Take a series with regime persistence, keep every value, and shuffle: the marginal distribution is identical but the predictability collapses. The information lived entirely in the ordering (§3 shows this: lag-1 autocorrelation $0.527\to 0.003$).
3. A recurrence is a learned sufficient statistic. Instead of feeding the model a hand-built window of lags, a recurrent model maintains a hidden state $h_t$ that summarises everything seen so far, and updates it as each new input arrives: $h_t=g(h_{t-1},x_t)$. This is the whole idea of a sequence model, and everything else (LSTM, GRU, Transformer) is a better-designed $g$.

The practical objective of this folder is then: choose $g$ (recurrence, attention, or convolution) and train it, while staying honest about a signal that is weak and a distribution that drifts.

---

2. Mathematical Ground Truth & Derivations

The IID assumption, written down. Standard supervised learning assumes $(x_t,y_t)$ are i.i.d. draws from a joint $p(x,y)$, so the optimal predictor is the conditional mean/expectation $\mathbb{E}[y\mid x]$ and the joint likelihood factorises:

$$p(x_1,y_1,\dots ,x_T,y_T)=\prod _{t=1}^{T}p(x_t,y_t).$$

The factorisation is the assumption. For a time series it is false in two ways - through the inputs and the targets:

$$p(x_1,\dots ,x_T)=\prod _{t}p(x_t\mid x_{t-1},x_{t-2},\dots ),p(y_t\mid x_{1:t})\ne p(y_t\mid x_t).$$

The Markov chain (the simplest sequence model). The cleanest tractable assumption is order-1 Markov:

$$p(x_t\mid x_{1:t-1})=p(x_t\mid x_{t-1}).$$

The state is just the previous value; the whole past is summarised by one number. This is already a sequence model, and it is the discrete cousin of the RNN: an RNN generalises “state $=$ previous value” to “state $=$ learned vector $h_{t-1}$”, with a continuous transition

$$h_t=g_{\theta }(h_{t-1},x_t),{\hat{y}}_t=f_{\varphi }(h_t),$$

and parameters $\theta ,\varphi$ fitted by maximising ${\sum }_t\log p(y_t\mid h_t)$. The Kalman filter is the special case where $g$ is linear-Gaussian and the state has a closed-form optimal update (see Kalman Filtering).

The value of memory, quantified. For a persistence process the autocovariance ${\gamma }_k=\mathrm{Cov}(x_t,x_{t+k})$ is positive and slowly decaying, so the lag-$k$ predictor’s skill is ${\rho }_k={\gamma }_k/{\gamma }_0$. A function of $x_t$ alone captures only ${\rho }_0=1$‘s instantaneous structure; a function of the recent path captures ${\rho }_k>0$. The next pages make $g$ expressive enough to capture these dependencies in both directions (past→future) and at multiple scales.

---

3. Computational Implementation - order carries information

Stdlib only. We generate a regime-switching return series (the sign persists, as in bull/bear phases), then keep every value but shuffle the order and measure what is lost: lag-1 autocorrelation, mean run length, and the accuracy of a simple recurrent predictor (“the running sum of the last $k$ returns predicts the sign of the next”).

Read the punchline: the shuffled and original series have identical values (identical mean, variance, histogram) yet the shuffled one is nearly a fair coin ($0.505$ accuracy, run length $2.02\approx$ the geometric baseline), while the original is highly predictable from a simple recurrent state ($0.797$ accuracy, run length $5.00$). The information is in the ordering, and a model with a state can read it. More importantly for finance: the recurrent predictor was two lines of arithmetic, not a deep net - the value is in having a state at all, which is the point the more elaborate architectures on the next pages build on.

---

4. Failure Modes & First-Principles Breakdowns

1. “Order doesn’t matter if I just use good features.” False in general. Hand-built lag features can approximate state, but they fix the window length in advance and cannot represent a state that is updated by new information. A recurrence learns the sufficient statistic instead of you guessing it.
2. “Shuffling is harmless” - the exact error that inflates CV scores in the sibling Purged CV folder. For a sequence model, shuffling is not merely a validation bug; it destroys the signal you are trying to learn (autocorr $0.527\to 0.003$ above).
3. Stationarity is smuggled in. The Markov/RNN view assumes the transition $g_{\theta }$ is stationary - the same rule at every $t$. Financial regimes violate this (page 05 quantifies it: a model fit on regime A has $R^2=+0.76$ in-sample but $-0.68$ out-of-regime). Order carries information; the rule generating that order does not stay fixed.
4. State can memorise noise. With enough hidden units, $h_t$ can encode idiosyncratic shocks that look predictive in-sample and mean nothing out-of-sample - the low-SNR failure developed in Financial ML Pitfalls.

---

5. References

• Goodfellow, Bengio & Courville, Deep Learning, Ch. 10 (§10.1–10.3: why sequence modeling needs state; the unrolled computational graph; the RNN as a learned sufficient statistic)
• Tsay, Ruey S., Analysis of Financial Time Series (3rd ed.)
• López de Prado, Advances in Financial Machine Learning, Ch. 1

---

6. Connected Graph Bridges

• Back: Index Hub · Financial ML Pitfalls & Low SNR
• Forward: 02 · RNNs & LSTMs
• Classical state-space cousin: Signal Processing & Kalman Filtering
• Features the state model replaces/consumes: Feature Engineering & Labeling