8.6.1 Data Infrastructure & Reproducibility from Zero

Basic Prerequisites: none beyond running a Python script; the example needs only the standard library.

---

1. Intuition & Practical Objective

This page builds the why of data infrastructure and reproducibility with no prior systems knowledge needed. The objective is one idea: an unpinned result is not a result - if the same code cannot be run again by anyone, anywhere, and produce the same bytes, then whatever a backtest or a paper reported is unverifiable and therefore scientifically worthless.

Start with the dumbest question: why does reproducibility even matter? A strategy has an edge only if it is true. If your pipeline silently produced slightly different data on Monday than on Tuesday - a different seed, a different library version, a different rounding - then a backtest that “works” may be reproducing your bug, not a real edge. The entire discipline of this folder is to make the difference between “the run that discovered the edge” and “the run that proves it” equal to exactly nothing.

Three steps:

1. “It ran yesterday” is not evidence. The default state of any software system is drift - a dependency upgrades, a data vendor restates a field, a float rounds differently on a new CPU. Evidence means the run can be re-executed today and compared byte-for-byte against the original. A checksum is the cheapest such comparison.

2. Determinism is a property you enforce, not one you assume. Random-number generators are deterministic only when seeded; floating-point arithmetic is deterministic only when the order of operations is fixed. Both silently default to “surprise me” unless you pin them. That pinning is the whole job of reproducibility tooling.

3. A checksum turns “trust me” into “check me.” Hashing a dataset produces a short, fixed fingerprint; if a single byte anywhere changes, the fingerprint changes. You do not need to look at the data to know it is intact - you compare two 64-character strings.

---

2. Mathematical Ground Truth & Derivations

A hash is a many-to-one map with astronomically rare collisions. A SHA-256 digest maps an arbitrary byte string to $\mathbf{h}\in \{0,1{\}}^{256}$. If two different datasets $D_1\ne D_2$ produced the same digest, we call it a collision; for a cryptographically strong hash the best attack still costs ~$2^{128}$ evaluations, so the chance of an accidental collision on two real datasets is effectively

$$\mathrm{Pr}[\text{collision}]\approx 2^{-256}\approx 8.6\times 10^{-78}.$$

That number is why a checksum is treated as proof of integrity, not a heuristic.

Hashing is linear in length. Digesting a dataset of byte length $L$ costs $O(L)$ time and $O(1)$ memory (the running state is fixed at 256 bits). This is what makes checksums cheap enough to run on every block of every pipeline.

Floating-point is not an exact arithmetic. IEEE-754 doubles keep about 16 significant decimal digits. When you sum values of very different magnitude, the small term can be absorbed - $1e16+1=1e16$ in double precision because the unit-in-the-last-place of $10^{16}$ is $2$. The order of a reduction therefore changes the result, and a parallel framework that reorders a sum can change the answer. This is not a bug in the math; it is the physics of finite precision, and reproducibility tooling must pin the order.

The reproducibility contract. Output $O$ is reproducible iff the triple (data version $D$, code commit $C$, environment lockfile $E$) is pinned:

$$O=f(D,C,E).$$

Change any one of the three inputs and you have, by definition, a different computation - even if it “looks the same.”

---

3. Computational Implementation - determinism you can touch

This is the single most convincing way to see the theory: generate a dataset twice from the same seed and prove with a checksum that the two runs are byte-identical, then show that a one-bit change in the seed makes the whole fingerprint change. Stdlib only (random.Random for reproducible seeding, hashlib.sha256 for the fingerprint).

Two lessons are printed literally: same inputs give a byte-identical digest (reproducibility is real and checkable), and one changed seed changes every byte of the fingerprint (a checksum is exquisitely sensitive to drift). This sensitivity is precisely what makes it a trustworthy guard: it never misses a change.

---

4. Failure Modes & First-Principles Breakdowns

1. The “it worked yesterday” trap. Without an explicit fingerprint of the inputs, you cannot tell whether today’s run used the same data, code, and environment as yesterday’s. The default is drift; reproducibility tooling is what makes drift visible.
2. Unseeded randomness is undiscoverable nondeterminism. random.random() without a fixed seed gives a different stream on every run. Two “identical” scripts can disagree and nothing raises an error - the divergence is silent.
3. Trusting statistics over integrity. A mean or a z-score looks plausible even when a column is subtly corrupted. Aggregates can hide single-row corruption; only a per-cell checksum sees a single flipped byte. (See the hub and 05 · Failure Modes.)

---

5. References

• López de Prado - Advances in Financial Machine Learning
• Astral uv / Poetry / conda-lock / Docker
• Needham & Simons - DuckDB in Action

---

6. Connected Graph Bridges

• Base: Quantitative Research (reproducibility is the precondition of trustworthy research)
• Continue: 02 · Data Pipelines · Index Hub
• Data origin: Data Sources & Corporate Data