7.6.3 Attention & Transformers

Basic Prerequisites: 02 · RNNs & LSTMs and linear algebra (dot products, softmax).

---

1. Intuition & Practical Objective

Recurrence compresses the past into one vector that is overwritten at each step. Attention does something different and powerful: it keeps all past states and lets the model look back and choose which ones to use. The mechanism is a soft dictionary lookup. Every past position offers a key $k_j$ and a value $v_j$; the current position issues a query $q_i$; the output is the similarity-weighted average of the values:

$${\text{output}}_i=\sum _j\underset{\text{softmax weight (attention)}}{\underbrace{\frac{\exp (q_i\cdot k_j)}{{\sum }_{j^{\prime }}\exp (q_i\cdot k_{j^{\prime }})}}}{v}_j.$$

Three “aha”s:

1. No fixed memory horizon. Attention’s path length between any two positions is $O(1)$ (they interact directly), versus $O(T)$ for recurrence. This is why Transformers learn long-range dependencies that RNNs cannot.
2. It is fully parallel over time. All queries and keys are computed at once - the reason Transformers train fast on GPUs, unlike the sequential BPTT of page 02.
3. Causality must be enforced, not assumed. In a forecasting task the query at $t$ must not see keys at $j>t$. A raw (bidirectional) attention will happily attend to the future - look-ahead leakage inside the architecture. The fix is an explicit causal mask $M$ (§3 verifies it produces an exact zero upper triangle).

The practical objective: read and write the attention formula correctly, and never build a forecasting model without a causal mask.

---

2. Mathematical Ground Truth & Derivations

2.1 Scaled dot-product attention (Vaswani et al. 2017)

Given a matrix of $T$ queries $Q\in {\mathbb{R}}^{T\times d_k}$, keys $K\in {\mathbb{R}}^{T\times d_k}$, values $V\in {\mathbb{R}}^{T\times d_v}$:

$$\boxed{\mathrm{Attn}(Q,K,V)=\mathrm{softmax}(\frac{Q{K}^{\top }}{\sqrt{d_k}}+M)V}$$

where $\mathrm{softmax}$ is applied row-wise and $M$ is the mask.

Why divide by $\sqrt{d_k}$? If the components of $q$ and $k$ are i.i.d. with mean $0$ and variance $1$, then $q\cdot k={\sum }_{m=1}^{d_k}{q}_m{k}_m$ has mean $0$ and variance $d_k$. Unscaled, the logits grow like $\sqrt{d_k}$, pushing the softmax into a saturated region where one weight $\to 1$ and the gradients $\to 0$. Dividing by $\sqrt{d_k}$ restores unit variance and keeps the softmax in its informative range (§3 measures this: score stdev $1.45\to 15.73$ as $d_k$ grows, and the softmax entropy collapses without the scaling).

2.2 The causal mask

For an autoregressive / forecasting model,

$$M_{ij}=\{\begin{array}{ll}-\infty & j>i\\ 0& j\le i\end{array}$$

so the pre-softmax logit for any future position is $-\infty$ and $\exp (-\infty )=0$. Position $i$ can attend only to $j\le i$. This is the exact attention analogue of the no-look-ahead requirement; without it, the model is trained on the answer.

2.3 Multi-head attention and the Transformer block

One attention head computes one similarity notion. Multi-head attention runs $h$ heads in parallel subspaces and concatenates:

$$\text{MultiHead}=\mathrm{Concat}({\text{head}}_1,\dots ,{\text{head}}_h)W^O,{\text{head}}_m=\mathrm{Attn}(Q{W}_m^Q,K{W}_m^K,V{W}_m^V).$$

A Transformer block adds residual connections and a position-wise feed-forward net: $z=\mathrm{LayerNorm}(x+\text{MultiHead}(x))$, $y=\mathrm{LayerNorm}(z+\mathrm{FFN}(z))$. Positional encodings (sinusoidal or learned) are added to the inputs, because attention - unlike a recurrence - is permutation-invariant and otherwise cannot tell position $5$ from position $50$.

---

3. Computational Implementation - causal attention and the scaling argument

numpy + stdlib. Part A computes masked attention and proves the causality property; Part B measures why the $\sqrt{d_k}$ scaling exists.

Read it as two facts. First, the weight matrix is lower-triangular by construction - the upper triangle is exactly zero, every row sums to $1$, and position $0$ can only attend to itself. This is what makes an attention forecasting model honest; drop the mask and you have built a leak. Second, the scaling matters: at $d_k=256$ the unscaled score standard deviation is $15.73$ and the softmax entropy collapses to $0.158$ (attention has pinned almost all weight on one key - gradient-starved), whereas the $\sqrt{d_k}$-scaled version stays at $1.75$, near the maximum-entropy value $\ln 8=2.08$ for $8$ positions.

---

4. Failure Modes & First-Principles Breakdowns

1. Missing causal mask = look-ahead leakage. The most common and most damaging bug. A bidirectional/encoder-only attention over a forecasting target sees the future by construction. Always verify $\mathrm{triu}(W,1)=0$ on a test batch. (The validation-side twin of this is the Purged CV folder.)
2. $O(T^2)$ cost in time and memory. Attention scales quadratically with sequence length; a full LOB day at tick resolution is millions of tokens. This is the practical reason microstructure models use local attention or convolution (DeepLOB) rather than a full Transformer.
3. Attention weights are not explanations. High attention on a timestep does not prove it caused the prediction; “attention is not explanation” (Jain & Wallace 2019). Use it as a diagnostic, not a story.
4. Positional encoding is not neutral in finance. Time is not a uniform grid (trading hours, overnight gaps, information-driven bars). Naive positional encoding imposes a regularity the market does not have.
5. Over-parameterisation on small data. A single Transformer block has millions of weights; on $N\sim 10^3$ financial samples it will memorise. Attention is a tool for the large-$N$ microstructure regime, not for monthly bars.

---

5. References

• Vaswani, Ashish et al. (2017), Attention Is All You Need, NeurIPS
• Lim, Bryan et al. (2021), Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting, IJF 37(4)
• Goodfellow, Bengio & Courville, Deep Learning, Ch. 10 (§10.4 “Encoder–Decoder Sequence-to-Sequence Architectures” and the attention mechanism)
• Zhang, Zohren & Roberts (2019), DeepLOB: Deep Convolutional Neural Networks for Limit Order Books, IEEE TSP 67(11), arXiv:1808.03668
• Jain, S. & Wallace, B. (2019), Attention is not Explanation

---

6. Connected Graph Bridges

• Back: 02 · RNNs & LSTMs · Index Hub
• Forward: 04 · Autoencoders for Factors
• Multi-horizon forecasting target: 06 · Advanced Extensions (TFT, DeepAR)
• Leakage discipline (validation side): Purged Cross-Validation & Backtest Hygiene
• Text/NLP attention (same mechanism, different data): Financial NLP & Transcripts