7.6 Deep Learning for Sequences

Basic Prerequisites: Calculus & Optimization (gradients, backpropagation) and Financial ML Pitfalls & Low SNR (why finance is a hard learning problem). The classical ancestor of every model here - the linear state-space / Kalman filter - lives at Signal Processing & Kalman Filtering. (these are the folder-level prerequisites for pages 02–06; page 01 states its own, smaller, entry requirements)

1. Intuition & Practical Objective

A cross-sectional model asks: given today’s features, what is tomorrow’s expected return? A sequence model asks a strictly harder question: given today’s features and the path that led here, what comes next? That extra word - the path - is the whole subject. Financial data arrives as an ordered stream (tick arrivals, order-book updates, bar returns, macro releases), and the ordering itself carries information that an order-blind model throws away.

This folder is the topic-hub for deep learning for sequential data in Kwant-Atlas. It (a) gives the fast architecture/formula lookup below - job #1 of a hub - and (b) routes you to six sub-pages that walk you from raw intuition through the recurrent, attention, and autoencoder families, the low-SNR reality that governs whether to use any of them, and the extensions (TCN, GRU, DeepAR, TFT, DeepLOB).

The one-sentence essence. “A sequence model is a learned state machine: a recurrence (RNN/LSTM) or an attention mechanism (Transformer, TCN) compresses the entire past into a state that predicts the future - but in finance the state is learned from a low-signal, non-stationary stream, so the binding constraint is almost never architecture, it is data and validation discipline.”

The honest framing up front, because it decides everything after it: deep sequence models are not automatically better than gradient-boosted trees in finance. On the medium-frequency tabular problems that dominate the industry, well-tuned trees match or beat deep nets (Gu, Kelly & Xiu 2020 tested both). Deep sequence models earn their place only where there is raw ordered structure with no good manual features - limit-order-book microstructure, high-frequency tick streams - where the sample size $N$ reaches $10^6$–$10^9$ and the sequence itself is the feature.

2. Mathematical Ground Truth & Derivations

Quick-Reference Lookup (job #1). Notation: $x_t$ input, $h_t$ hidden state, $C_t$ LSTM cell state, $\sigma$ logistic sigmoid, $\odot$ Hadamard (element-wise) product, $W/U$ recurrent/input weight matrices, $d_k$ attention head dimension, $M$ causal mask, $L$ layers, $K$ filter width. All checks below were re-executed and reproduced exactly (see §3).

Critical decision caveat (AFML Ch. 1; Gu–Kelly–Xiu 2020). The lookup table tells you how to build these models, not whether to. Before reaching for any row above, check the data budget: with $N$ effective (non-overlapping) samples and $p$ parameters you need $N\gg p$, and financial returns have an effective sample size far smaller than the calendar count because of serial correlation and non-stationarity. Section 05 turns this into a decision rule.

3. Computational Implementation - the architecture probe

Runs on numpy + stdlib only. It reproduces the four headline numbers in §2 in a single pass: the vanishing-gradient gap between a vanilla RNN and an LSTM, the causal property of masked attention, and the TCN receptive field.

Read it as the folder’s thesis in numbers: the vanilla RNN’s gradient from 50 steps ago is $\sim 10^{-27}$ (vanished - it cannot learn long-range dependencies at all), while the LSTM’s cell gradient is still $0.605$ (the gates keep the memory channel open). Attention sidesteps the recurrence entirely and builds a strictly causal, row-normalised weighted average. The TCN reaches 8 steps into the past with only 3 layers and 2-wide filters.

4. Failure Modes & First-Principles Breakdowns

Hub signposts - the decision rule and practice checklist live in 05 · Failure Modes & Practice. In one line each:

1. Data hunger vs low SNR - deep nets carry $10^4$–$10^8$ parameters; that needs an effective sample count finance rarely supplies, so they overfit where trees do not ($\S$05).
2. Non-stationarity - the data-generating process shifts (regimes, arbitrage decay); a sequence model trained on regime A can have negative $R^2$ on regime B ($\S$05).
3. Look-ahead leakage - attention and recurrence both invite subtle future-peeking (no causal mask, or target leakage); this is the same disease the Purged CV folder treats, in architectural clothing ($\S$03).
4. Opacity & unhedgeable model risk - a black-box sequence model is hard to reason about when it breaks, and breaks silently under regime change ($\S$05).

5. References

• Goodfellow, Bengio & Courville: Deep Learning, MIT Press (2016)
• Hochreiter, Sepp & Schmidhuber, Jürgen: Long Short-Term Memory, Neural Computation 9(8):1735–1780 (1997)
• Vaswani et al.: Attention Is All You Need, NeurIPS (2017)
• Bai, Kolter & Koltun: An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling, arXiv:1803.01271 (2018)
• Lim, Arık, Loeff & Pfister: Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting, IJF 37(4):1748–1764 (2021)
• Heaton, Polson & Witte: Deep learning for finance: deep portfolios, Applied Stochastic Models in Business and Industry 33(1):3–12 (2017)
• Zhang, Zohren & Roberts: DeepLOB: Deep Convolutional Neural Networks for Limit Order Books, IEEE Trans. Signal Processing 67(11):3001–3012 (2019), arXiv:1808.03668
• López de Prado, Marcos: Advances in Financial Machine Learning (Wiley, 2018)
• Hastie, Tibshirani & Friedman: The Elements of Statistical Learning (2nd ed., 2009)

6. Connected Graph Bridges

• Foundational base: Calculus & Optimization · Probability & Measure Theory · Financial ML Pitfalls & Low SNR
• Classical sequence model (the linear ancestor): Signal Processing & Kalman Filtering - the Kalman filter is a state-space sequence model with a closed-form update; the RNN is its nonlinear, learned cousin.
• Feature side (what the sequence model replaces or consumes): Feature Engineering & Labeling
• Sibling (same pillar): Tree-Based Factor Ranking & Purged CV · Purged Cross-Validation & Backtest Hygiene · Deep Learning for Sequences
• Sub-pages (in-folder): 01 From Zero · 02 RNNs & LSTMs · 03 Attention & Transformers · 04 Autoencoders for Factors · 05 Failure Modes · 06 Advanced Extensions

Beginner: start at 01 · Practitioner: start at 05