4.8 Model Risk & Validation

Basic Prerequisites: Statistics & Inference (hypothesis testing, likelihood ratios, bias–variance) and Probability & Measure Theory. (these are the folder-level prerequisites for pages 02–06; page 01 states its own, smaller, entry requirements)

1. Intuition & Practical Objective

Every other folder in this pillar tells you how to compute a risk number. This one tells you what to do when the computation is wrong - which, as Derman put it, it always is at some level. Model risk is the risk of being wrong: the potential for adverse consequences from decisions based on incorrect or misused model outputs (SR 11-7, 2011).

The practical objective of this folder is to make the residual between the model and reality a first-class, measurable, managed quantity. Three working definitions anchor everything:

Model risk (regulatory, SR 11-7). “The potential for adverse consequences from decisions based on incorrect or misused model outputs and reports.” It arises for two reasons: (1) a model may have fundamental errors and produce inaccurate outputs relative to its design objective; (2) a model may be misused - applied outside its assumptions, or with a misunderstanding of its limitations.
Model risk (practitioner, Derman 1996). A taxonomy of seven failure channels, from “modeling is simply inapplicable” through “wrong inputs” to “software and hardware bugs”. Derman’s line - “you’re worse off thinking you have a model and relying on it than in simply realizing there isn’t one” - is the discipline’s founding sentence.
Effective challenge. SR 11-7’s governing principle: critical analysis by objective, informed parties able to identify limitations and force changes. It requires incentives, competence, and influence - an organisational property, not a spreadsheet.

The one-sentence essence. “A model is a toy: it forces you to state assumptions you cannot test, and the risk you carry is the sensitivity of the decision to those assumptions - so validation is the business of hunting the assumption the answer depends on, not of confirming the answer you wanted.”

This folder is a hub: it holds the fast lookup for validation statistics and the model-risk taxonomy below, and routes you to six sub-pages that walk from raw intuition through the sources of model risk, the validation and backtesting machinery, model-risk governance, the failure modes, and the advanced (uncertainty-quantification) extensions.

2. Mathematical Ground Truth & Lookup

Notation. $p$ = model’s nominal tail probability ( $p = 1 - α$ , e.g. $0.01$ for $99%$ VaR); $n$ = number of backtests; $x$ = number of exceptions (hits); $\overset{π}{^} = x / n$ ; $χ_{1}^{2}, χ_{2}^{2}$ the chi-square laws; $z_{1 - p} = Φ^{- 1} (1 - p)$ .

(a) The model-risk error decomposition. The gap between a model’s output and the “true” value is a sum of independent error channels, each separately quantifiable to first order:

total model error Δ V \approx wrong model Δ_{form} V + parameter/calibration i \sum \frac{\partial V}{\partial θ _{i}} Δ θ_{i} + discretisation, bugs, day-count Δ_{impl} V + misuse Δ_{use} V .

For small independent errors the channels add in quadrature: $σ_{total} = \sum_{j} σ_{j}^{2}$ . This is what converts “model risk” from a worry into a budget.

(b) Validation test statistics (VaR backtesting, $99%$ / $n$ ).

Quantity	Formula	Verified check (§3)
Kupiec POF (unconditional coverage)	$L R_{u c} = - 2 ln \frac{( 1 - p ) ^{n - x} p ^{x}}{( 1 - π ^ ) ^{n - x} π ^ ^{x}}$	$n = 250, x = 3 \Rightarrow L R_{u c} = 0.0949, p -val = 0.7580$
Christoffersen (independence)	$L R_{in d} = - 2 ln \frac{L _{indep} ( π ^ )}{L _{Markov} ( π ^ _{01} , π ^ _{11} )}$	$n = 250$ : $L R_{in d} = 0.0732$
Conditional coverage	$L R_{cc} = L R_{u c} + L R_{in d} \sim χ_{2}^{2}$	$L R_{cc} = 0.1681, p -val = 0.9194$
$χ_{1}^{2}$ tail p-value	$P (χ_{1}^{2} > z) = erfc (z /2)$	-
$χ_{2}^{2}$ tail p-value	$P (χ_{2}^{2} > z) = e^{- z /2}$	-

(c) The Basel traffic-light zones (BCBS 1996, $99%$ / $250$ days). The zone boundaries are binomial quantiles of the hit count under true $99%$ coverage; the yellow zone begins where $P (K \leq x) \geq 95%$ , the red where $\geq 99.99%$ .

Exceptions $x$	$L R_{u c}$	$p$ -value	$P (K \leq x ∣ 99%)$	Zone	Plus	$k = 3 + plus$
0–4	≤3.84 $^{†}$	≥0.05	-	green	0.00	3.00
5	1.9568	0.1619	0.958817	yellow	0.40	3.40
6	3.5554	0.0594	0.986299	yellow	0.50	3.50
7	5.4970	0.0190	0.995975	yellow	0.65	3.65
8	7.7336	0.0054	0.998943	yellow	0.75	3.75
9	10.2290	0.0014	0.999750	yellow	0.85	3.85
≥10	≥12.96	≤0.0003	≥0.999946	red	1.00	4.00

Cross-check: the computed binomial $P (K \leq 5 ∣ 99%) = 0.958817$ reproduces the $95.88%$ printed in BCBS (1996) §2 - the boundary that fixes yellow at five exceptions.

(d) Uncertainty-quantification lookups (advanced, §06).

Quantity	Formula
Bayesian model weight (BIC)	$w_{i} \propto e^{- \frac{1}{2} Δ BIC_{i}}$
BMA estimate	$\overset{ˉ}{V} = \sum_{i} w_{i} V_{i}$ , model-selection sd $= \sum_{i} w_{i} (V_{i} - \overset{ˉ}{V})^{2}$
KL divergence (Gaussian)	$D_{KL} (P ∥ Q) = ln \frac{σ _{Q}}{σ _{P}} + \frac{σ _{P}^{2} + ( μ _{P} - μ _{Q} ) ^{2}}{2 σ _{Q}^{2}} - \frac{1}{2}$
Entropy-robust price bound	$sup_{Q : D_{KL} (Q ∥ P) \leq ε} E_{Q} [X] \approx E_{P} [X] + 2 ε Var_{P} (X)$

3. Computational Implementation - the traffic-light / Kupiec engine

Standard library only. Reproduces every number in the lookup tables. kupiec is the likelihood-ratio; binom_cdf recovers the BCBS zone boundaries; plus is the regulatory multiplier add-on.

4. Failure Modes & First-Principles Breakdowns

Hub signposts - the full analysis lives in 05 · Failure Modes & Practice. In one line each:

Overfitting to validation data. Once a validation set is used to select a model, it stops being a validation set; the reported error is biased low by the selection. Reuse of the judge corrupts the trial.
Benchmark error. A comparison is only as good as its benchmark; if the benchmark shares the model’s flaw (same data, same bias), it certifies a wrong answer. Benchmark correlation must be measured, not assumed away.
Model drift / regime change. A model calibrated in one regime keeps returning a number while the world moves out from under it - coverage decays silently until the exceptions cluster.
The model risk of VaR itself. A $99%$ VaR is an estimate carrying sampling error ( $\sim 0.23 σ$ at $n = 250$ ); validation tests inherit that noise, so “passing” a backtest is weak evidence, and the test statistic itself is a model.

5. References

Derman, E. - Model Risk, Goldman Sachs Quantitative Strategies Research Notes (April 1996). The founding taxonomy: three meanings of “model”; the seven types of model risk; the seven guidelines for avoidance. Read in full from the corpus PDF (44_Derman_1996_model_risk.pdf).
Board of Governors of the Federal Reserve System & OCC - Supervisory Guidance on Model Risk Management, SR Letter 11-7 / OCC 2011-12 (April 2011). The binding definitions of model, model risk, effective challenge, the three validation elements (conceptual soundness, ongoing monitoring, outcomes analysis) and the governance framework. Read in full from the corpus PDF (45_OCC_2011…).
Morini, M. - Understanding and Managing Model Risk: A Practical Guide for Quants, Traders and Validators (2011, Wiley). The rare quantitative treatment of model uncertainty, validation and limits, written by a working validator.
Tunaru, R. - Model Risk in Financial Markets: From Financial Engineering to Risk Management (2015, World Scientific). Broad academic–industry survey of model-risk measurement.
BCBS - Supervisory Framework for the Use of Backtesting in Conjunction with the Internal Models Approach to Market Risk Capital Requirements (1996, BIS). The three-zone traffic light and the capital multiplier.
Hastie, Tibshirani & Friedman - The Elements of Statistical Learning, 2nd ed. (2009)

6. Connected Graph Bridges

Foundational base: Statistics & Inference · Probability & Measure Theory · Bayesian Statistics
Sub-pages (in-folder): 01 From Zero · 02 Sources of Model Risk · 03 Validation & Backtesting · 04 Model-Risk Management · 05 Failure Modes · 06 Advanced Extensions
Sibling topics: Value at Risk & Expected Shortfall · Parametric, Historical & Monte Carlo VaR · Extreme Value Theory & Fat Tails · Stress Testing & Scenario Analysis
Forward: Financial ML Pitfalls & Low SNR (backtest overfitting, purged CV) · 06 · Advanced Extensions (uncertainty quantification)

Beginner: start at 01 · Practitioner: start at 05

Kwant Atlas

Explorer

4.8 Model Risk & Validation

1. Intuition & Practical Objective

2. Mathematical Ground Truth & Lookup

3. Computational Implementation - the traffic-light / Kupiec engine

4. Failure Modes & First-Principles Breakdowns

5. References

6. Connected Graph Bridges

4.8.1 Model Risk from Zero

4.8.2 Sources of Model Risk

4.8.3 Validation & Backtesting

4.8.4 Model-Risk Management

4.8.5 Failure Modes & Practice

4.8.6 Advanced Extensions