2.6.2 Why Execution Backtests Lie

Basic Prerequisites: 01 · From Zero.

---

1. Intuition & Practical Objective

An execution backtest lies for the same reason a weather forecast lies: it substitutes a convenient model for a hard process, and the substitution is systematically optimistic. This page names the five lies an execution backtest tells and shows, in first-principles terms, which direction each one biases the answer.

The five lies, in one line each:

1. The fill illusion - a trade printed at my price, so I filled (wrong: you fill only when the outflow ahead of you passes $\xi \ge x$).
2. The queue blind spot - I assume I am at the front of every queue (wrong: $x$ is often most of the level’s volume).
3. Look-ahead in replay - I decide the fill using flow that arrives after I would have cancelled.
4. The zero-cost instant - I act instantly and at the mid (wrong: latency and spread are real).
5. The invisible impact - my order does not move the market (wrong: it does, and it is the cost that grows fastest with size).

The one-sentence essence. “Every execution-backtest bias is an unmodelled fill: the backtest fills more often, earlier, and at better prices than the process allows - and because fills are convex in the P&L, the errors compound.”

---

2. Mathematical Ground Truth & Derivations

The truth the naive rule replaces. Under price-time (FIFO) priority, your profit from a resting order of size $L$ at queue position $x$ is a function of the cumulative outflow ahead $\xi$:

$$\text{Filled}(x,L,\xi )=(\xi -x{)}^{+}-(\xi -x-L{)}^{+},(z{)}^{+}=\max (z,0).$$

The naive rules are special, biased, cases:

Backtest rule	Mathematical statement	Direction of bias
”any print”	$1[\xi >0]$	up (fills when $\xi <x$ too)
“trades only, ignore queue”	$1[{\xi }_{\text{trade}}\ge L]$	up (drops $x$)
ignore cancels ahead	$\xi =\text{trades}$	down (understates $\xi$)
look-ahead	$\xi =\xi (T_{\text{day}})$ instead of $\xi (t_{\text{cancel}})$	up (uses the future)
fill at mid	$p=m$ instead of $p=m\pm c$	up

Fill probability. For Poisson trade arrivals at rate $\mu$, the outflow by time $T$ is $\xi (T)\sim \text{Poisson}(\mu T)$, so an order at the front ($x\to 0$) is filled by $T$ iff at least one trade arrives, and an order at position $x$ (in shares) satisfies (with $\mu$ the share-volume rate)

$$\mathbb{P}(\text{reach front by }T)=\mathbb{P}\left(\xi (T)\ge x\right)=1-\sum _{k<x}\frac{(\mu T{)}^k{e}^{-\mu T}}{k!},$$

a Poisson upper-tail (equivalently, the waiting time is negative-binomial with mean $x/\mu$).

Slippage. Define slippage per share relative to the arrival (decision) price ${\pi }_0$ as

$$\text{slip}=\underset{\text{what you paid}}{\underbrace{(p-{\pi }_0)}}-\underset{\text{unavoidable move}}{\underbrace{(m_T-{\pi }_0)}},$$

so that a fill at $p=m_t$ with no impact gives $\text{slip}=m_t-m_T$ - the adverse post-fill drift that a naive test never charges. Averaged, $\mathbb{E}[\text{slip}\mid \text{filled}]=\text{AS}>0$ (a cost).

Look-ahead. A replay that decides the fill at the end of the day, when a live order would have been cancelled at $t_c$, computes $\mathbb{P}(\xi (T_{\text{day}})\ge x)$ instead of $\mathbb{P}(\xi (t_c)\ge x)$. Since $\xi$ is non-decreasing, the look-ahead probability is $\ge$ the causal one, with equality only when $t_c=T_{\text{day}}$.

---

3. Computational Implementation - the five lies, quantified

A single simulated level (queue ahead 11,500 shares; my size 800) replayed under five fill rules. Stdlib only.

Read the table as the anatomy of the illusion: the optimistic rule fills every path ($1.0000$); the queue-blind rule also fills every path (my 800 shares are trivially covered by $11,500$ shares of flow); the correct FIFO rule fills only $0.3686$ of paths and, when it does, only $29\%$ of the size on average. Feeding that into a simple edge model ($0.02 capture, $0.012 adverse selection per share) turns a naive $16.00 per order into a realistic $1.86 - the edge is overstated by $8.6\times$ before a single other bias is added.

---

4. Failure Modes & First-Principles Breakdowns (numbered)

1. The fill illusion (optimistic fills). Replacing $1[\xi \ge x]$ with $1[\xi >0]$ makes the backtest fill on paths where you are still behind the queue. Measured: $1.0000$ vs $0.3686$. Fix: always express a fill as a condition on $\xi$ relative to $x$ and $L$.
2. The queue blind spot. Dropping $x$ means the backtest cannot see the difference between “at the front” and “10,000 shares deep”; both fill at $1.0000$ here. Fix: track $x$ per level, per venue, inclusive of the effect of your own repricing.
3. Look-ahead in replay. Using end-of-window flow to decide a fill made at the cancel time. Fix: replay must be causal - only events with timestamp $\le$ decision time may affect the fill.
4. The zero-cost instant. Assuming fills at the mid and zero latency deletes the spread ($p=m\pm c$) and the pick-off term $\mathbb{P}\approx 1-e^{-\rho \ell }$. Fix: charge half-spread per leg, model latency explicitly (Low-Latency Systems Architecture).
5. The invisible impact. Assuming $p={\pi }_0$ drops $m_{t+5}-m_t$, the impact term you caused. Fix: calibrate an impact model (Almgren 2005) and charge it as a function of size and participation.
6. Compounding. These biases are not additive noise - they multiply. A fill-rate overstatement of $2.7\times$ combined with a zero-cost assumption of $2\times$ can turn a flat strategy into a “Sharpe 2” backtest. This is the bridge to Backtesting Hygiene: overfitting and execution bias are two independent inflation engines, and they stack.

---

5. References

• Hasbrouck, Joel - Empirical Market Microstructure (2007)
• Cont, Rama; Kukanov, Arseniy - “Optimal order placement in limit order markets,” Quantitative Finance 17(4) (2017)
• Gould et al. - “Limit order books,” Quantitative Finance 13(11) (2013)
• López de Prado, Marcos - Advances in Financial Machine Learning (2018)
• Perold, A. F. - “The implementation shortfall” (1988)

---

6. Connected Graph Bridges

• Back: 01 · From Zero · Index Hub
• Forward: 03 · The Fill Model
• Depth: Queue Position & Fill Probability · Backtesting Hygiene