4.8.5 Failure Modes & Practice

Basic Prerequisites: 03 · Validation & Backtesting and 04 · Model-Risk Management.

---

1. Intuition & Practical Objective

The validation machinery of 03 is only as good as the process around it. This page names the four failure modes that defeat a technically correct framework, each with the first-principles reason and a runnable demonstration:

1. Overfitting to validation data (selection bias). Once a validation or backtest set is used to choose a model, its error estimate is biased low - sometimes spectacularly. The validator becomes a trainer, and the trial loses its meaning.
2. Benchmark error. A benchmark that shares the model’s flaw certifies the same wrong answer. Independence must be demonstrated, not assumed.
3. Model drift / regime change. A model calibrated in one regime keeps returning a confident number while coverage decays silently. The number is stale but looks live.
4. The model risk of VaR itself. The risk measure is an estimate with sampling error; the validation tests inherit that error. “Passing” a backtest is weak evidence, and the test is itself a model.

The unifying principle. Every one of these is a multiple-comparisons or non-stationarity failure: either you looked at the data too many times (1, 2, 4), or the data-generating process moved (3). Recognising which one you face tells you the fix.

---

2. Mathematical Ground Truth & Derivations

2.1 Selection bias (the expected maximum). Suppose you screen $M$ genuinely skill-less models, each giving an estimate ${\hat{t}}_i\sim N(0,1)$ (a $t$-statistic under the null). If you keep the best, its reported statistic is ${\max }_i{\hat{t}}_i$, whose expectation grows like

$$\mathbb{E}\left[\underset{i\le M}{\max }{\hat{t}}_i\right]\approx \sqrt{2\ln M}-\frac{\ln \ln M+\ln 4\pi }{2\sqrt{2\ln M}}\stackrel{M\uparrow }{\to }+\infty .$$

So “the best of $M$” is by construction large even when nothing is real. Equivalently, to keep a family-wise error rate $\alpha$ you must compare against the Bonferroni threshold $z^{\star }={\Phi }^{-1}(1-\alpha /M)$, which exceeds the naive ${\Phi }^{-1}(1-\alpha )=1.96$ by a widening margin.

2.2 The deflated Sharpe ratio. Bailey & López de Prado convert 2.1 into a corrected significance test. With $T$ observations and $M$ trials, the sharpest $t$-statistic must beat $\sqrt{2\ln M}$ (the expected maximum under the null) before it is evidence at all. The probability of backtest overfitting is the rate at which the in-sample best fails out-of-sample; it rises with $M$ and falls with $T$.

2.3 Calibration slippage under drift. If the model is calibrated on a period with volatility ${\sigma }_0$ but the world moves to ${\sigma }_1>{\sigma }_0$, the true exception rate becomes

$${\pi }_1=\Phi \left(-\frac{{\sigma }_0}{{\sigma }_1}{z}_{1-p}\right)(\text{for normal tails, }{z}_{1-p}={\Phi }^{-1}(1-p)).$$

For $p=0.01$, $z_{0.99}=2.326$: at ${\sigma }_1=2{\sigma }_0$, ${\pi }_1=\Phi (-1.163)=0.122\to$ the expected exception rate is $\approx 12\%$ - a $12\times$ breach with no change in the code. This is exactly the number measured in the drift demonstration below.

2.4 The estimator’s own error. A $99\%$ VaR estimated from $n$ observations carries density-quantile standard error $\mathrm{se}\approx \frac{1}{f(q)}\sqrt{\alpha (1-\alpha )/n}$, which shrinks only as $n^{-1/2}$ ($\approx 0.23\sigma$ at $n=250$, per the VaR hub). Both the reported VaR and the backtest that grades it are randomised by this - an under-appreciated source of spurious green zones.

---

3. Computational Implementation

(A) Selection bias - “the best of 1000 models”. Generate $M=1000$ strategies, each $252$ days of pure noise (true skill zero), and screen by $t$-statistic. Stdlib only, seeded.

The best of $1000$ pure-noise strategies reports $t=3.64$ - a marginal $p$-value of $0.00014$, which naively reads as “highly significant”. It fails the Bonferroni threshold $z^{\star }=3.8906$. Meanwhile $32$ of the $1000$ clear the naive $1.96$ bar and $14$ clear $2.58$ - all of them noise. The theoretical $\sqrt{2\ln M}$ ($3.72$ at $M=1000$) tracks the simulated maximum ($3.64$) closely. This is why “we validated on a held-out set” is not sufficient: if the held-out set was used to select, it is no longer held out.

(B) Model drift. A $99\%$ VaR model calibrated on a calm regime ($\sigma =1$) against a world that shifts to $\sigma =2$ for the last $100$ days.

The same model, unchanged, goes from $0.80\%$ to $12.0\%$ exceptions - a $15\times$ degradation - exactly when a doubled volatility arrives. The closed form of §2.3 predicts ${\pi }_1=\Phi (1.163)\approx 12\%$; the simulation returns $12.0\%$. Drift is not a code bug and will not be caught by re-running tests on old data; it is caught by ongoing monitoring and by the regime-dependence of the estimates themselves.

---

4. Failure Modes & First-Principles Breakdowns

1. Validation-set reuse → biased low. Every peek at the validation set spends a degree of freedom; the reported error falls below the truth (ESL §7.1: “the test set is kept in a vault”). Fix: nested/purged CV, a true holdout, or an explicit multiple-testing correction (Bonferroni, deflated Sharpe).
2. The “many-model” illusion of robustness. Screening $M$ models and picking the winner manufactures apparent skill (§3A). Fix: pre-register the model, or report the deflated statistic, or size the search budget into the evidence.
3. Benchmark error. A benchmark sharing data/code/structure with the model cannot challenge it (03 · §3B showed $DM=-0.04$, indistinguishable). Fix: benchmark on different data or a different model class, and measure the correlation of their errors.
4. Drift and non-stationarity. Models are conditional on a regime; when it changes, coverage decays silently (§3B). Fix: ongoing monitoring, rolling recalibration, volatility filtering (FHS/GARCH), EVT tails, and stress backtests.
5. The estimator’s noise mistaken for signal. With $n=250$ the $99\%$ VaR has $\approx 0.23\sigma$ sampling error; two desks with identical risk report VaRs a quarter-$\sigma$ apart, and a single year’s backtest cannot distinguish them. Fix: report confidence bands on the VaR and on the test, and require economic (not just statistical) significance.
6. Gaming the validator. Optimising for the backtest (e.g., inflating VaR slightly, or widening the window) passes the test without improving the model - Goodhart’s law applied to validation. The cure is SR 11-7’s effective challenge: a party with the incentive and standing to ask “why did this change?“

---

5. References

• Hastie, Tibshirani & Friedman, ESL 2nd ed. (2009)
• Bailey, D. & López de Prado, M., The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality, Journal of Portfolio Management 40(5) (2014)
• Federal Reserve / OCC, SR 11-7 (2011) - “Validation is an important check during periods of benign economic and financial conditions, when estimates of risk and potential loss can become overly optimistic”; ongoing monitoring and benchmarking. Read in full from the corpus PDF.
• Derman, E., Model Risk (Goldman Sachs QSR, 1996) - “A model may be reasonable, but the world itself may be unstable… a good model today may be inappropriate tomorrow.” Read in full from the corpus PDF.
• Christoffersen, P. (1998)
• McNeil & Frey (2000)

---

6. Connected Graph Bridges

• Back: 04 · Model-Risk Management · 03 · Validation & Backtesting
• Forward: 06 · Advanced Extensions (BMA, KL, robust bounds) · Index Hub
• Sibling: Financial ML Pitfalls & Low SNR (backtest overfitting in ML) · Value at Risk & Expected Shortfall (estimation error)