7.7.1 Reinforcement Learning for Trading from Zero

Basic Prerequisites: Probability Theory (conditional expectation).

---

1. Intuition & Practical Objective

This page builds the why of reinforcement learning with no prior RL knowledge needed. The objective is one idea: some financial decisions are not predictions but actions whose consequences change the situation you will face next - and those are exactly the problems a supervised model cannot solve.

Start with the dumbest question: why isn’t a return forecast enough? A supervised model answers “what will the price do?” That is a prediction. But a trader’s problem is “given what I hold and where I am in the day, how much do I trade now?” That is a decision. The two differ because the decision feeds back:

1. The action changes the state. Sell 5,000 shares now and the price moves; the price you get for the next 5,000 is worse. A forecaster treats the market as a fixed backdrop; the trader’s own trades are part of the market for as long as the order is working.

2. The reward is delayed and path-dependent. You cannot score a single trade in isolation. The value of selling slowly is that you avoid impact (good now) but carry risk (bad later). Judgment requires a sequence of actions, judged by the total.

3. Only trial-and-error can find the policy. There is no labelled dataset of “the optimal number of shares to sell at 10:03 for this inventory.” You have to try, observe the P&L, and adjust. That loop - act, observe reward, update policy - is reinforcement learning.

Three “aha”s.

1. RL is goal-directed control, not prediction. The object learned is a policy $\pi (a\mid s)$: a mapping from situations to actions. It is not a forecast; it is a plan you can execute.
2. The reward function is the whole game. You do not get credit for a “clever” algorithm. You get credit for the objective you encoded. Get the reward wrong and a brilliant optimizer will faithfully maximize the wrong thing (05 · Failure Modes).
3. Context matters - “always do X” is rarely optimal. The optimal action depends on the state. This is why even a one-step trading decision, done right, is state-dependent.

---

2. Mathematical Ground Truth & Derivations

The decision loop, formally. At each time step $t$ the agent observes a state $S_t$, picks an action $A_t\sim \pi (\cdot \mid S_t)$, receives reward $R_{t+1}$, and lands in a new state $S_{t+1}$. The agent’s goal is to maximize the expected discounted return

$$G_t=\sum _{k=0}^{\infty }{\gamma }^k{R}_{t+k+1},\gamma \in [0,1).$$

The discount $\gamma$ is not a technicality - in trading it is the time value of waiting: a dollar of P&L realized later is worth less, and for an execution problem $\gamma$ encodes how much you fear the price wandering while your order is unfilled.

The value of acting - the action-value function. The natural object is $Q^{\pi }(s,a)$, the expected return of taking action $a$ in state $s$ and then following policy $\pi$:

$$Q^{\pi }(s,a)={\mathbb{E}}_{\pi }\left[G_t\mid S_t=s,A_t=a\right].$$

A policy is better if its $Q$ is larger; the best possible policy obeys the Bellman optimality equation

$$Q^{\ast }(s,a)=R(s,a)+\gamma \sum _{s^{\prime }}P(s^{\prime }\mid s,a)\underset{a^{\prime }}{\max }{Q}^{\ast }(s^{\prime },a^{\prime }),$$

and the optimal action in any state is simply $a^{\ast }=\arg {\max }_a{Q}^{\ast }(s,a)$. This one equation is the target every value-based method in 03 · Value-Based RL chases. Policy-gradient methods in 04 reach the same objective by climbing the gradient of expected return instead.

Why the Markov assumption is a modelling choice, not a gift. The Bellman equation is exact only if $S_t$ summarizes everything relevant about the past - the Markov property. In trading you engineer this: inventory, time-to-deadline, realized volatility, imbalance. If your state omits something the reward depends on (e.g. the unfilled part of your order), the MDP is misspecified and no algorithm saves you. (Compare the Markov-switching state of HMM & GMM.)

---

3. Computational Implementation - the smallest possible trading MDP

The most convincing way to see “state-dependent policy beats a fixed rule”: a one-step contextual decision with three market states. In each state the agent may trade (earn a noisy edge ${\alpha }_s$, pay a cost $c$) or hold (earn $0$). The exact optimum is “trade iff ${\alpha }_s>c$,” but RL has to discover which states those are by sampling. Stdlib only.

The learned policy matches the exact optimum (trade in states 0 and 2, hold in state 1) and the learned $Q$ values track the true edge minus cost ($+0.30-0.20\approx +0.10$, $+0.55-0.20\approx +0.35$, $-0.10-0.20\approx -0.30$), with the gap being sampling noise. The state dependence is the whole point: “always trade” would have traded in state 1 and destroyed value. RL found the conditional rule from experience.

---

4. Failure Modes & First-Principles Breakdowns

1. The “it’s just a fancy classifier” trap. RL is not supervised learning with a different loss. If your problem has no action that affects the next state, use supervised learning - RL only adds variance and complexity. If it does feed back (impact, inventory, risk limits), supervised learning is structurally wrong.
2. Reward misspecification. The agent maximizes exactly what you wrote. Reward raw P&L and the agent learns to trade aggressively (ignoring impact); reward trade count and it learns to churn. The objective you intend must be the objective you encode - see 05 · Failure Modes.
3. The Markov-state illusion. “More features” is not the same as “enough state.” If the reward depends on hidden variables (queue position, unfilled size, regime), the learned policy is optimizing a misspecified world. This is the RL analogue of omitted-variable bias.
4. Discounting as a hidden risk preference. $\gamma$ silently sets how much the agent fears the future. Set it carelessly and you have smuggled in a risk attitude you never chose; it interacts directly with the risk-aversion of Almgren–Chriss (Almgren–Chriss).

---

5. References

• Sutton, Richard S. & Barto, Andrew G.: Reinforcement Learning: An Introduction (2nd ed., 2018)
• Bertsekas, Dimitri P.: Dynamic Programming and Optimal Control, Vol I
• Nevmyvaka, Feng & Kearns (ICML 2006)
• López de Prado, Marcos: Advances in Financial Machine Learning

---

6. Connected Graph Bridges

• Base: Probability & Measure Theory · Calculus & Optimization
• Sibling: Regime Classification: HMM & GMM (the Markov state of the market) · Financial ML Pitfalls & Low SNR
• Continue: 02 · The MDP Framing · Index Hub
• Execution context: Optimal Execution & Almgren–Chriss