1.2.5 Failure Modes & Real-World Practice

Basic Prerequisites: 04 · The Deflated Sharpe Ratio and 02 · Why Backtests Lie.

---

1. Intuition & Practical Objective

Pages 01–04 named the diseases; this page is the clinic. The objective: a concrete, ordered protocol that a researcher runs before a backtest is allowed to influence a capital decision - and a numbered catalogue of the ways otherwise-good analyses fail. The discipline is not cynicism; it is knowing exactly where the number is soft so the residual can be measured and gated.

The theme of every failure mode: a forgiving assumption applied at a place where the analyst benefits from the forgiveness. Costs are assumed zero where they bite; $N$ is undercounted where it inflates; the sample is chosen after the outcome. Each is a first-principles violation of an exchangeability or an accounting identity, not a matter of taste.

---

2. Mathematical Ground Truth & Derivations

2.1 The Sharpe cost drag (why a few basis points flip a verdict)

If the strategy turns over fraction $\tau$ of its book per year and faces round-trip cost $c$ (proportional spread + fees), the annual cost drag is $\tau c$, so the net Sharpe is

$${\widehat{SR}}_{\text{net}}={\widehat{SR}}_{\text{gross}}-\frac{\tau c}{{\sigma }_{\text{ann}}}\text{(annualized, }{\sigma }_{\text{ann}}\text{ as annual vol).}$$

The drag scales as $\tau /{\sigma }_{\text{ann}}$: high-frequency, low-vol strategies are crushed by costs that are irrelevant to slow, high-vol ones. This is not a modelling nicety - it is the single most common reason a Sharpe-2.0 backtest becomes a Sharpe-0.7 live strategy. Market impact adds a square-root term $\propto (\text{size}/\text{ADV}{)}^{1/2}$ that makes the drag size-dependent (Almgren–Chriss); a flat “5 bps” assumption is itself a failure mode.

2.2 Effective $N$ and the non-independence trap

Trials produced by tweaking one base signal are correlated; the selection threshold uses $\hat{N}\approx \hat{\rho }(M-1)+1$ (page 03). Reporting the raw sweep $M$ without $\hat{\rho }$ is uninterpretable in both directions. Guideline: report the full sweep $M$, an estimated average correlation $\hat{\rho }$, and the implied $\hat{N}$.

2.3 The decision gate

A defensible gate combines three independent corrections, all of which must pass:

$$\underset{\text{selection bias + non-normality}}{\underbrace{\text{DSR}\ge 0.95}}\wedge \underset{\text{accounting}}{\underbrace{{\widehat{SR}}_{\text{net}}>0\text{after full costs}}}\wedge \underset{\text{sample sufficiency}}{\underbrace{T\ge \text{MinTRL}}}\wedge \underset{\text{selection robustness (page 06).}}{\underbrace{\text{PBO}\ll 0.5(\text{e.g.}\le 0.05)}}$$

Note the recursion: computing DSR needs $N$, which needs the search to have been logged - so the protocol must be enforced during research, not audited after.

2.4 The order of operations (hygiene checklist)

1. Pre-register the hypothesis and the plan for the search; log every configuration (this fixes $\hat{N}$).
2. Point-in-time everything - universe, fundamentals, prices, index membership.
3. Split by time (never randomly) into IS/tuning and a vaulted OOS, used once.
4. Model costs fully: spread, fees, borrow, and size-dependent impact, calibrated to your turnover.
5. Compute ${\widehat{SR}}_{\text{net}}$, skew, kurtosis, $\hat{N}$, $V[\{{\widehat{SR}}_n\}]$.
6. Gate on DSR, MinTRL, and PBO; report all three.
7. Freeze the chosen configuration; any post-hoc change resets the OOS - the vault is spent.

---

3. Computational Implementation - when costs kill the strategy

A 5-year, high-turnover strategy with a true gross annual Sharpe of 2.0, found among $N=20$ trials. We apply proportional costs and recompute both the net Sharpe and the DSR (trial-SR dispersion 0.5 annualized). Stdlib only.

A gate that passes at zero cost fails at 2 bps - a cost level no high-turnover strategy can honestly assume away. The lesson is not “costs bad”; it is that selection-bias correction and cost accounting are not independent filters - a corrected Sharpe and an uncorrected one live on opposite sides of the threshold, and both must be computed at the same point in the protocol.

---

4. Failure Modes & First-Principles Breakdowns (numbered)

1. The missing $N$ (selection bias). A backtest without the trial count is uninterpretable (Bailey & López de Prado). First principle: the reported statistic is a maximum, which has no meaning without the number of draws.
2. Transaction-cost amnesia. Assuming zero or fixed-dollar costs rather than proportional spread + quadratic impact. Symptom: live Sharpe collapses and is negative at the assumed turnover; the drag scales as $\tau /{\sigma }_{\text{ann}}$.
3. Repeated holdout. Using the “out-of-sample” set for many rounds of selection turns 5% false positives into expected ones (~20 applications). Fix: one vaulted use.
4. Survivorship / point-in-time failure. Backtesting today’s constituents; using restated data. First principle: membership required past success - a selection filter baked into the sample.
5. Look-ahead via vintages. Fundamentals, index adds/drops, or vendor-adjusted prices known only later. Symptom: unnaturally smooth equity curves.
6. Effective-$N$ blindness. Treating 1,000 correlated variants as 1,000 independent bets (or as one). Report $M$, $\hat{\rho }$, and $\hat{N}$.
7. Non-normality ignored. Quoting Sharpe for a negatively-skewed, fat-tailed stream (carry, option-selling) whose true standard error is far larger - DSR’s denominator exists precisely for this.
8. Capacity & alpha-decay blindness. A strategy can be statistically real yet too small to matter or decayed by the time it is deployed; neither DSR nor PBO knows about capacity.
9. Metric gaming. Sharpe on illiquid mark-to-model returns, or on smoothed series, understates variance and inflates the ratio - a data-quality door, not a statistics door.
10. Publishing bias in the literature itself. The documented “anomalies” you mine for ideas are the survivors of other people’s searches; their prior inflation is inherited, not reset (White 2000; Harvey, Liu & Zhu’s “most claimed findings are likely false”).

---

5. References

• Bailey, D. H. & López de Prado, M.: The Deflated Sharpe Ratio (2014)
• Harvey, C. R. & Liu, Y.: Backtesting, JPM (2015)
• Harvey, C. R., Liu, Y. & Zhu, H.: … and the Cross-Section of Expected Returns, RFS 29(1) (2016)
• Almgren, R. & Chriss, N.: Optimal Execution of Portfolio Transactions, Journal of Risk (2001)
• Hastie, Tibshirani & Friedman: The Elements of Statistical Learning (2nd ed.)

---

6. Connected Graph Bridges

• Back: 04 · Deflated Sharpe · Index Hub
• Forward: 06 · Purged CV, PBO & Reality Check
• Siblings: Feature Engineering & Labeling (triple-barrier labels determine the purge width) · Stat-Arb & Pairs Trading (turnover-heavy, cost-sensitive)
• Cross-pillar: Financial ML Pitfalls & Low SNR