A.8.5 Failure Modes & Practice

Basic Prerequisites: 01 · From Zero through 04 · Insider & Ownership.

---

1. Intuition & Practical Objective

Every dataset in this folder is a survivor of its own history, and if you read it naively you inherit the future as if it were the past. This page is the defense layer: it takes the four ways fundamental data lies to a backtest - point-in-time bias, survivorship bias, restatement gaps, and API/coverage limits - and ties each to a first principle, then quantifies each so the bias is a number, not a warning. If the hub table is the map, this page is the compass check.

The deepest principle: a dataset is a function of when you looked at it. Today’s database is $\mathcal{D}(t_{\text{now}})$, and it encodes every deletion, restatement, and backfill since inception. A backtest that reads $\mathcal{D}(t_{\text{now}})$ for a decision made at time $t<t_{\text{now}}$ is not merely imprecise - it is reading information flow from the future, and the resulting performance is a measurement of the leak, not of the strategy.

---

2. Mathematical Ground Truth & Derivations

Survivorship bias. Let the true universe return be the average over all firms that existed at time $t$, $\bar{r}=\frac{1}{N}{\sum }_{i=1}^N{r}_i$, and the survivorship-filtered average be over the sub-universe that still exists today, ${\bar{r}}_{\text{surv}}=\frac{1}{M}{\sum }_{i\in \text{live}}{r}_i$ with $M<N$. The bias is

$${\text{Bias}}_{\text{surv}}={\bar{r}}_{\text{surv}}-\bar{r}=\frac{1}{M}\sum _{i\in \text{live}}{r}_i-\frac{1}{N}\sum _{i=1}^N{r}_i>0,$$

strictly positive because the delisted firms are, on average, the losers. It is largest for value and distress screens - the very strategies whose premise is that some firms do not survive.

Point-in-time / look-ahead bias. With $\mathcal{D}(t)$ the vendor’s database as of $t$, an honest signal is a function only of the past:

$$\text{honest: }{\sigma }_t=f(\mathcal{D}(t))\text{vs.}\text{leaky: }{\sigma }_t=f(\mathcal{D}(t_{\text{now}})).$$

Any strategy that replaces $\mathcal{D}(t)$ with $\mathcal{D}(t_{\text{now}})$ has substituted the restated, backfilled, survivor-filtered record for what was actually knowable - the arithmetic cost of which is precisely the restatement gap in 01 §2.

Restatement gap. For a line item $x$ with as-filed value $x_f$ and final restated value $x_r$,

$$\text{gap}=\frac{x_r-x_f}{x_f},\text{and the sign of the gap is not random:}$$

restatements cluster on the downside (write-offs, revenue reversals), so a backtest built only on restated $x_r$ overstates how bad things looked in real time and, symmetrically, overstates how predictable the trouble was.

API / coverage arithmetic. Pulling a universe of $N$ firms one request at a time at the SEC’s advised ceiling $r=10$ req/s costs at least $N/r$ seconds:

$$t_{\min }=\frac{N}{r}=\frac{5000}{10}=500\text{s}\approx 8.3\text{minutes}\text{before throttling, retries, or backoff.}$$

The bulk/frames route collapses that to a handful of downloads - the practical difference between a pipeline and a denial-of-service against yourself.

---

3. Computational Implementation - the three biases, measured

Offline and fully runnable (stdlib only). It quantifies survivorship bias, the restatement look-ahead, and the API-cost gap in one pass.

A 21.25-point survivorship gap on a five-firm toy universe is deliberately stark, but the direction and mechanism scale: the firms that vanished are the ones that went to zero, and any screen worth running is one that would have held some of them. The look-ahead case is even sharper - the firm the screen picked (highest as-reported ROE, Y) is the firm that was later restated negative. A leaky backtest would book Y as a winner; an honest one holds the write-off.

The practice checklist (data hygiene as a build gate).

1. Key everything by $(firm,concept,period,\text{filing date})$ - never overwrite on restatement.
2. Lag every source by its legal deadline (Form 4: 2 days; 13-F: 45 days; 10-Q: 40–45 days; 10-K: 60–90 days) before it can enter a signal.
3. Include delisted firms and their terminal returns; build the universe from historical index membership, not today’s tickers.
4. Reconcile across at least two sources before trusting a line item (03).
5. Pull bulk, not per-name, and cache to disk with the request date stamped on it.

---

4. Failure Modes & First-Principles Breakdowns

The catalog, each tied to its first principle.

1. Point-in-time bias (first principle: a dataset is a function of when you looked). Using $\mathcal{D}(t_{\text{now}})$ for a decision at $t$. Symptom: backtest returns that collapse when you move from a vendor’s “as-reported” to “point-in-time” file. Fix: Compustat PIT, EDGAR as-filed vintages, or a self-built keyed panel.
2. Survivorship bias (first principle: the dead are the signal). Screening today’s live tickers. Symptom: a value/distress screen that looks extraordinary but holds no firm that ever failed. Fix: historical universe construction including delistings and terminal returns.
3. Restatement gaps (first principle: revisions are non-random and downside-clustered). A database keeping only the latest value, or a comparison across a restatement boundary without flagging it. Symptom: a line-item history that silently changes when the vendor refreshes. Fix: version rows by filing date; flag any period whose value moved.
4. Look-ahead via 13-F / ownership (first principle: the legal lag bounds knowledge). Treating “current institutional ownership” as known today when it is a Q-end snapshot filed 45 days later. Fix: lag by filing date (04).
5. API / coverage limits (first principle: rate limits are a first-class design constraint). Naive per-company loops; assuming a free tier covers the whole universe (it usually samples). Symptom: truncated universes, silent 429s, IP blocks. Fix: bulk endpoints, caching, exponential backoff, and an identifying User-Agent.
6. Vendor-field drift (first principle: schemas are contracts). A provider renames or restandardizes a field and your pipeline silently reinterprets history. Fix: a canonical schema + a provider-provenance column, plus a reconciliation test that fails loudly when a number moves.
7. Time-zone and fiscal-calendar ambiguity (first principle: “the quarter” is not universal). A June-FYE firm’s “Q4” is not December; a filing timestamped after market close is knowable only the next day. Fix: store fiscal period explicitly and stamp the next tradable time as the knowable-at date.

---

5. References

• WRDS, Compustat Point-in-Time documentation and the Compustat User’s Guide
• SEC, EDGAR APIs
• Kenneth R. French, Data Library and its description files
• Sloan, Richard (TAR, 1996) and Dechow, Sloan & Sweeney (TAR, 1995)
• Fama & French (JF, 1992; JFE, 2015)

---

6. Connected Graph Bridges

• Back: 02 · SEC EDGAR & XBRL · 03 · Commercial Providers · 04 · Insider & Ownership · Index Hub
• Forward: 06 · Advanced Extensions
• Defense layer: Accounting Quality & Red Flags (restatements as shenanigans) · Quantitative Fundamental Investing (the backtests these rules protect)