7.7 Reinforcement Learning for Trading

Basic Prerequisites: Probability & Measure Theory and Calculus & Optimization. (these are the folder-level prerequisites for pages 02–06; page 01 states its own, smaller, entry requirements)

1. Intuition & Practical Objective

Supervised learning predicts a label; reinforcement learning (RL) chooses an action and lives with the consequence. In trading this is the correct abstraction for everything sequential and path-dependent: how much to buy now given the inventory you still must liquidate, the impact you have already caused, and the risk you are carrying. Unlike a return forecast, an execution or portfolio decision changes the state you will face next - sell too fast and you pay impact, sell too slow and you carry price risk. RL is the branch of machine learning that optimizes exactly this closed-loop, state-dependent control problem.

This folder is the machine-learning pillar’s control-theory topic-folder. It is a hub: it (a) gives you the fast algorithm lookup below (job #1 of the topic), and (b) routes you to six sub-pages that build the intuition from zero, formalize the MDP, derive value-based and policy-gradient methods, name the failure modes honestly, and land on the one place RL has genuinely earned its keep in finance - optimal trade execution.

The one-sentence essence. “Model trading as a Markov decision process - state (inventory, prices, time), action (how much to trade), reward (P&L net of costs) - and learn a policy that maximizes expected discounted reward; the hard parts are not the algorithms (Q-learning and policy gradients are a page of math) but the reward design, the sim-to-real gap, non-stationarity, and the fact that finance gives you one non-repeatable episode instead of a million resettable ones.”

Honesty clause (read before deploying anything). RL is the most over-promised technique in quantitative finance. The results that survive scrutiny are narrow and specific - chiefly execution / order-scheduling under a known impact model (Nevmyvaka & Kearns 2006; Ning, Lin & Jaimungal 2018). End-to-end “RL beat the market” claims almost always fail out of sample because the simulator is wrong, the reward is gameable, and the market adapts. This folder teaches the machinery and the reasons it usually disappoints.

2. Mathematical Ground Truth & Derivations

Quick-Reference Lookup (job #1). The Bellman equations and the four core updates. Notation: $S$ state, $A$ action, $R$ reward, $\gamma \in [0,1)$ discount, $\alpha$ learning rate, $\tau$ temperature, $\theta$ policy parameters, ${\pi }_{\theta }(a\mid s)$ policy, $V^{\pi }$, $Q^{\pi }$, $V^{\ast }={\max }_{\pi }{V}^{\pi }$.

Verified checks (from §3, stdlib only). Value iteration on the liquidation MDP gives $V^{\ast }=(0,7.000,12.300,16.070)$ with greedy policy $[0,1,1,1]$; a tabular Q-learning agent recovers it to $0.0000$ max error; the Bellman optimality residual on the stochastic execution MDP is $0.00\times 10^0$; a REINFORCE agent converges to the same greedy policy $[0,1,1,1]$.

The one caveat that matters. All three families (DP, TD/Q-learning, policy gradient) are solving the same Bellman fixed point. If they disagree, the bug is in the reward or the state, not in the optimizer - which is why §4 and 05 · Failure Modes spend most of their time on reward and state design, not on algorithms.

3. Computational Implementation - the Bellman engine

This runs on the standard library only. It is the mental template for the whole folder: define a small trading MDP, solve it exactly with value iteration, and remember this number - every RL agent in the sub-pages must match it.

Note the non-monotone optimal action: with $x$ shares left the agent sells one at a time even though selling all at once is available - convex impact ($k{a}^2$) makes gradual liquidation optimal. This is precisely the trade-off that Almgren–Chriss solves in closed form (Optimal Execution & Almgren–Chriss) and that an RL agent must rediscover from experience.

4. Failure Modes & First-Principles Breakdowns

Hub signposts - the full treatment lives in 05 · Failure Modes & Practice. In one line each:

1. Reward hacking - the agent optimizes the proxy you wrote, not the objective you meant. A gross-P&L reward ignores impact, so the agent churns (verified: net value $9.00$ vs the correct $16.07$, a $7.07$ loss).
2. The sim-to-real gap - the agent overfits the simulator’s micro-behaviour (fills, latency, impact). Your backtest environment is the model, and it is wrong.
3. Non-stationarity - a policy tuned to one regime (drift, vol, impact) decays in the next; the market is an adversary that arbitrages away your edge (verified: a stale drift policy loses $0.52$ immediately on a sign flip).
4. Sample inefficiency - finance offers one historical path and no resets; deep RL needs $10^5$–$10^7$ episodes. Simulation on a wrong impact model is the standard, and usually fatal, workaround.
5. Backtest overfitting of the policy itself - an RL agent with a large state/action space is an extremely flexible model; it will memorize noise unless evaluated with purged CV / deflated Sharpe (Financial ML Pitfalls · 05).

5. References

• Sutton, Richard S. & Barto, Andrew G.: Reinforcement Learning: An Introduction (2nd ed., MIT Press, 2018)
• Bertsekas, Dimitri P.: Dynamic Programming and Optimal Control, Vol I–II
• Nevmyvaka, Yuriy; Feng, Yi & Kearns, Michael: “Reinforcement Learning for Optimized Trade Execution” (ICML 2006)
• Bertsimas, Dimitris & Lo, Andrew W.: “Optimal Control of Execution Costs” (Journal of Financial Markets 1(1), 1998)
• Moody, John & Saffell, Matthew: “Learning to Trade via Direct Reinforcement” (IEEE Trans. Neural Networks 12(4), 2001)
• Mnih, Volodymyr et al.: “Human-level Control through Deep Reinforcement Learning” (Nature 518, 2015)
• Schulman, John et al.: “Proximal Policy Optimization Algorithms” (arXiv:1707.06347, 2017)

6. Connected Graph Bridges

• Foundational base: Probability & Measure Theory (Markov chains, conditional expectation) · Calculus & Optimization (gradients, fixed points) · Econometrics & Time Series (Markov structure, non-stationarity)
• Sibling topics (in-pillar): Deep Learning for Sequences (the function approximators RL policies are built on) · Regime Classification: HMM & GMM (Markov state) · Financial ML Pitfalls & Low SNR (why the backtest lies)
• Execution & cost cousins: Optimal Execution & Almgren–Chriss · POV · Constraints & Transaction Costs
• Sub-pages (in-folder): 01 From Zero · 02 The MDP Framing · 03 Value-Based RL · 04 Policy-Gradient & Actor-Critic · 05 Failure Modes & Practice · 06 Advanced Extensions

Beginner: start at 01 · Practitioner: start at 05