7.7.3 Value-Based RL

Basic Prerequisites: 02 · The MDP Framing and Deep Learning for Sequences.

---

1. Intuition & Practical Objective

Value-based RL learns how good each action is and then acts greedily. It never needs a model of the market: instead of predicting the next state’s probability (as value iteration does), it bootstraps - it uses its own current estimate of the value of the next state to improve the current estimate, watching only the realized reward and the realized next state. The celebrated update is Q-learning, and its deep version is DQN.

Practical objective: understand why Q-learning works (it is stochastic approximation of the Bellman optimality operator), when it is the right tool in trading (small, discrete action spaces - execution, sizing, regime switching), and why the finance setting stresses it to breaking (non-stationarity, tiny samples).

The key idea in one line: $Q(s,a)\leftarrow Q(s,a)+\alpha [r+\gamma {\max }_{a^{\prime }}Q(s^{\prime },a^{\prime })-Q(s,a)]$ - move the estimate of the chosen action toward the observed reward plus the discounted best value you currently believe is reachable.

---

2. Mathematical Ground Truth & Derivations

2.1 Temporal-difference learning

The TD error ${\delta }_t=R_{t+1}+\gamma V(S_{t+1})-V(S_t)$ is the surprise between the bootstrapped target and the current estimate. TD(0) nudges the value by $\alpha {\delta }_t$:

$$V(S_t)\leftarrow V(S_t)+\alpha [R_{t+1}+\gamma V(S_{t+1})-V(S_t)].$$

TD is a stochastic approximation of the Bellman expectation backup; it converges to $V^{\pi }$ for a fixed policy under decaying step sizes (Robbins–Monro conditions $\sum {\alpha }_t=\infty ,\sum {\alpha }_t^2<\infty$).

2.2 Q-learning and the max operator

Off-policy Q-learning (Watkins & Dayan 1992) approximates the optimality operator regardless of the behaviour policy:

$$\boxed{Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\alpha [R_{t+1}+\gamma \underset{a}{\max }Q(S_{t+1},a)-Q(S_t,A_t)]}$$

The $\max$ is what makes it off-policy (it learns about the greedy policy while behaving $ϵ$-greedily). For tabular Q-learning with $ϵ$-greedy exploration and decaying $\alpha$, $Q\to Q^{\ast }$ with probability 1 (Watkins & Dayan). SARSA is the on-policy variant, using the actually taken next action $Q(S_{t+1},A_{t+1})$ instead of the max - safer under risk (it accounts for exploratory mistakes) but converges to a different, more conservative policy.

2.3 Why the max overestimates - and why that matters in finance

$\mathbb{E}[{\max }_{a}Q]\ge {\max }_a\mathbb{E}[Q]$ (Jensen). Because Q-learning maximizes over noisy estimates, it systematically overestimates values - the maximization bias. In trading, overestimation bias becomes over-trading: the agent believes actions are better than they are and acts when it should hold. Fixes: Double Q-learning (decouple action selection from evaluation), and its deep form Double DQN. This bias is first-principles, present in any maximization over noisy value estimates, and is a major reason naive RL agents trade too much.

2.4 DQN: value-based RL with function approximation

When the state space is large (continuous price features, portfolios), a table is impossible; DQN (Mnih et al. 2015) approximates $Q(s,a;\theta )$ with a neural network and stabilizes the divergence-prone “deadly triad” (function approximation + bootstrapping + off-policy) using two tricks:

• Experience replay - store transitions $(s,a,r,s^{\prime })$ in a buffer and sample minibatches, breaking temporal correlation.
• Target network - a slowly-updated copy $Q(\cdot ;{\theta }^{-})$ supplies the bootstrap target, freezing a moving goalpost.

Loss: $L(\theta )={\mathbb{E}}_{(s,a,r,s^{\prime })\sim \mathcal{D}}\left[\right(r+\gamma {\max }_{a^{\prime }}Q(s^{\prime },a^{\prime };{\theta }^{-})-Q(s,a;\theta ){)}^2]$.

Finance caveat: replay assumes stationarity - the same $(s,a)$ keeps its value across time. That assumption is precisely what markets violate, so replay buffers that span regimes silently mix incompatible data (05 · Failure Modes).

---

3. Computational Implementation - tabular Q-learning recovers $Q^{\ast }$ exactly

The single most convincing demonstration: run Q-learning with no knowledge of $P$ or $R$ (only sampled transitions) on the liquidation MDP from the hub, and show it converges to the value-iteration solution. Stdlib only.

Q-learning, seeing only $(x,a,r,x^{\prime })$ samples, reproduces the exact $V^{\ast }$ to $0.0000$ and the exact greedy policy $[0,1,1,1]$. This is the promise of value-based RL: learn the optimal control from experience, without a model. Note both hardcoded facts it needed - exploration that decays (otherwise it never visits and refines the good actions) and a step-size schedule that decays (otherwise the estimates never settle). Remove either and the result degrades; that sensitivity is the practical cost of being model-free.

---

4. Failure Modes & First-Principles Breakdowns

1. Maximization bias → over-trading. The $\max$ in Q-learning inflates values under noise (Jensen), so the learned policy acts too often. Use Double Q-learning / Double DQN. In a market with costs, “acting a little too often” is a direct, measurable P&L leak.
2. Exploration is dangerous in trading. $ϵ$-greedy exploration executes real trades in production. Either explore only in simulation, or replace random exploration with safe exploration (small perturbations around a known-good baseline, e.g. TWAP).
3. The deadly triad. Function approximation + bootstrapping + off-policy can diverge. DQN’s replay and target network are stability hacks, not proofs; the failure modes (semi-gradient divergence, Tsitsiklis & Van Roy) are real, especially with a non-stationary target.
4. Replay assumes stationarity. A huge replay buffer mixes transitions from different regimes; the network learns an average that is optimal for none. Time-limited buffers or regime-stratified replay help but do not solve it.
5. Sample inefficiency. Tabular Q-learning is cheap only because it is tiny. DQN needs $10^5$–$10^7$ transitions; finance gives one path. The usual workaround - train in a simulator - imports the simulator’s biases wholesale (the sim-to-real gap, 05).
6. Discrete-action limitation. Q-learning needs a $\max$ over a finite action set. Continuous sizing / portfolio weights require either discretization (coarse, and the grid itself is a modelling error) or the policy-gradient methods of 04.

---

5. References

• Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed., 2018)
• Watkins, Christopher & Dayan, Peter: “Q-learning” (Machine Learning 8, 1992)
• van Hasselt, Hado; Guez, Arthur & Silver, David: “Deep Reinforcement Learning with Double Q-learning” (AAAI 2016)
• Mnih, Volodymyr et al.: “Human-level Control through Deep Reinforcement Learning” (Nature 518, 2015)
• Sutton, Richard S.: “Learning to Predict by the Methods of Temporal Differences” (Machine Learning 3, 1988)
• Deng, Yue et al.: “Deep Direct Reinforcement Learning for Financial Signal Representation and Trading” (IEEE TNNLS, 2017)

---

6. Connected Graph Bridges

• Back: 02 · The MDP Framing
• Forward: 04 · Policy-Gradient & Actor-Critic · 05 · Failure Modes · Index Hub
• Function approximators: Deep Learning for Sequences (LSTM/TCN/Transformer value networks)
• Practical application: Optimal Execution & Almgren–Chriss · 06 · Advanced Extensions (Execution RL)