7.5.5 Failure Modes & Practice

Basic Prerequisites: 03 · The Pipeline and 04 · Evaluating Signal.

---

1. Intuition & Practical Objective

Every alt-dataset is a survivor of its own history, and a naive backtest inherits the future as if it were the past. This page quantifies the four ways alt-data lies to a backtest - backfill bias, survivorship bias, panel/coverage drift, and look-ahead - each tied to a first principle and each turned into a number rather than a warning. It is the defense layer that makes the pipeline of page 03 and the protocol of page 04 trustworthy.

The deepest principle is the same one Data Sources · 05 states for fundamentals: a dataset is a function of when you looked at it. Today’s vendor database $\mathcal{D}(t_{\text{now}})$ encodes every backfilled value, deleted entity, and restated number since inception. A backtest that reads $\mathcal{D}(t_{\text{now}})$ for a decision made at $t<t_{\text{now}}$ is not imprecise - it is reading information flow from the future, and the resulting performance measures the leak, not the strategy.

---

2. Mathematical Ground Truth & Derivations

Backfill bias. A vendor assigns missing early history a value that was unknown at the time (“backfilling”), or overwrites it with a later correction (“reinstatement”) - AFML §2.2.1. Let the as-originally-known value be $x_t^{\text{pit}}$ and the backfilled value $x_t^{\text{bf}}=x_t^{\text{pit}}+c_t$, where the correction $c_t$ is informed by the outcome $y_t$ (that is what a correction is). Then

$${\text{IC}}_{\text{bf}}=\mathrm{corr}(x^{\text{pit}}+c,y)>\mathrm{corr}(x^{\text{pit}},y)={\text{IC}}_{\text{pit}}\text{because }\mathrm{cov}(c,y)>0.$$

The inflation is largest in the oldest data - exactly where backfill is strongest and where a long backtest “proves” the signal.

Survivorship bias. Let the true universe average be over all entities that existed at $t$, $\bar{r}=\frac{1}{N}{\sum }_{i=1}^N{r}_i$, and the vendor universe average be over today’s survivors, ${\bar{r}}_{\text{surv}}=\frac{1}{M}{\sum }_{i\in \text{live}}{r}_i$ with $M<N$:

$${\text{Bias}}_{\text{surv}}={\bar{r}}_{\text{surv}}-\bar{r}>0,$$

strictly positive because the delisted firms are, on average, the losers. For alt-data this bites twice: the entity universe may be survivor-filtered, and the panel itself (which stores a vendor covers) is survivor-selected.

Panel / coverage drift. A card panel’s observed spend is $S_t^{\text{obs}}={\alpha }_t{S}_t^{\text{true}}$ where ${\alpha }_t$ = the panel’s coverage share. If an issuing bank leaves at $t^{\star }$, $\alpha$ drops discontinuously yet $S^{\text{true}}$ is unchanged, so the signal reads

$$\frac{S_{t^{\star }+1}^{\text{obs}}}{S_{t^{\star }}^{\text{obs}}}=\frac{{\alpha }_{t^{\star }+1}}{{\alpha }_{t^{\star }}}\approx 0.7(\text{a fake }-30\%\text{ "collapse"}),$$

i.e. vendor attrition masquerades as a demand shock - and it is the single most famous alt-data false signal. The fix: model the share (relative to a control) or normalize by panel size each period.

Look-ahead (restated once more, because it is the most common). Using $t_E$ instead of $t_K$ (page 03) or the final restated value at its first release date both inject future information; the resulting IC is inflated by exactly the leaked covariance.

---

3. Computational Implementation - the two headline biases in numbers

numpy only. Part A injects a realistic backfill (early values revised toward the outcome) and measures the IC inflation; Part B simulates a firm universe where weak firms delist and measures the survivorship bias in the mean return.

Read both. Backfill lifts the measured IC from $+0.4509$ to $+0.6069$ - a $+0.1561$ inflation - and the gap is nearly 50% larger in the old half ($+0.2313$ vs $+0.1561$ overall), exactly where a long backtest finds its “evidence.” Survivorship adds +2.95%/yr of phantom return: the true universe earned $+6.43\%$ but today’s survivors earned $+9.38\%$, so a backtest on the survivor universe manufactures almost half again the true return out of nothing. Both biases are pure bookkeeping errors - and both flatter the strategy at precisely the moments you would have gotten killed live.

---

4. Failure Modes & First-Principles Breakdowns (numbered)

1. Backfill & reinstatement time-travel (AFML §2.2.1). Vendors overwrite history with corrections; storing only the latest value changes your past whenever you re-run. Keep multiple knowledge-dated versions; forbid backwards edits.
2. Survivorship, in the universe and the panel. Delisted names and dropped panel members are the losers; excluding them is not “clean data,” it is a selection on the outcome. Quantified above at $+2.95\%$/yr.
3. Panel / coverage drift read as demand. Vendor attrition (${\alpha }_t$ changing) fakes a $-30\%$ collapse overnight. Model share or normalize by panel size; never trust a level you didn’t coverage-adjust.
4. Look-ahead by event-time tagging. The $t_K$ vs $t_E$ mistake (page 03) is the most common of all; it silently doubles an IC. Testable before any modelling.
5. Overfitting the evaluation. Searching $K$ datasets and reporting the best $t$-stat ignores that the winner is selected on noise; deflate for $K$ (deflated Sharpe / PBO). The evaluation is a search and must be corrected like one.
6. Compliance / MNPI. Ingesting web-scraped data that violates ToS, or a feed that embeds material non-public information (e.g. an unresolved earnings number inferred from a leaked transcript), exposes the fund to insider-trading liability. First principle: alt-data must be built from public or properly-licensed primary sources; “we found it on the web” is not a legal basis for trading it.

---

5. References

• López de Prado, Advances in Financial Machine Learning
• Luo, Yin et al.: “Seven Sins of Quantitative Investing” (Deutsche Bank, 2014)
• Bailey, López de Prado, et al.: “The Probability of Backtest Overfitting” and “The Deflated Sharpe Ratio”
• Guida, Tony, Big Data and Machine Learning in Quantitative Investment (Wiley, 2019)
• In-repo companion: Data Sources · 05 Failure Modes & Practice

---

6. Connected Graph Bridges

• Back: 04 · Evaluating Signal · Index Hub
• Forward: 06 · Advanced Extensions
• Sibling failure analysis: Financial ML Pitfalls & Low-SNR · Purged CV & Backtest Hygiene
• Provenance discipline: Data Sources & Corporate Data · Backtesting Hygiene