8.2.5 Failure Modes & Real-World Practice

Basic Prerequisites: 04 · Zero-Cost Abstraction.

---

1. Intuition & Practical Objective

High-performance C++ fails in a small, recurring set of ways, and every one of them has a first-principles cause you can name in advance. This page is the fault catalogue: the five ways a fast-looking trading path quietly becomes slow or wrong, each tied to the mechanism (§2) that produces it, plus the practical discipline that prevents it.

The five failures, one line each:

1. Hidden allocation in the hot path - a push_back/make_unique/std::function allocates, and the tail (not the mean) explodes.
2. Cache misses - a locality-blind layout fetches $3\times$ (or more) the bytes it uses, starving on memory latency.
3. Undefined behaviour (UB) - signed overflow, aliasing violations, and data races don’t just “give weird results”; they let the optimiser delete the code you measured, so the benchmark lies.
4. Branch misprediction - unpredictable control flow flushes a 14–20 stage pipeline (~15–20 cycles) per miss; data ordering alone can eliminate it.
5. The micro-optimisation fallacy - tuning arithmetic while the real cost is memory layout (§2.2) or the allocator (§2.3).

One-sentence essence. “In a latency path, ‘correct’ and ‘fast’ are the same requirement: any hidden allocation, cache miss, or UB is simultaneously a performance defect and a correctness risk.”

---

2. Mathematical Ground Truth & Derivations

2.1 Amortised vs bounded growth (the reallocation spike)

A doubling vector appending $N$ elements copies (element-wise) a total of

$$C(N)=\sum _{k=0}^{\lfloor {\log }_{2}N\rfloor }{2}^k\approx N-1,$$

so the amortised cost per append is $C(N)/N\approx 1$ element copy - $O(1)$ in the mean. But the worst single append, when $\text{size}=\text{capacity}$, copies $\text{capacity}\approx N/2$ elements in one operation, at a moment chosen by the data, not by you. The gap defines the engineering rule:

$$C_{\text{reserve}}(N)=0\text{copies},\text{reallocations}=0⟹\text{bounded }O(1)\text{ per append.}$$

Amortised $O(1)$ optimises the mean; the hot path needs bounded $O(1)$ to protect the tail. reserve() converts the first into the second.

2.2 Cache traffic (recap of the layout law)

For an $s$-byte AoS record scanning one field, distinct 64-byte lines scale as $L\propto N\min (1,s/64\text{ rounded up in the period})$; for the 24-byte order of 03 · Memory & Cache it is exactly $L_{\text{AoS}}/L_{\text{SoA}}=3$. Each extra line is a potential DRAM miss ($\sim 250$ cycles) that the arithmetic was not responsible for. Cache misses are the failure mode that looks like nothing is wrong in the source.

2.3 Branch misprediction

A modern CPU speculates past branches; a correct guess is free, a wrong guess flushes the pipeline for $r\approx 15\text{–}20$ cycles. For a branch taken with probability $p$, a simple predictor is wrong with probability $m$ - for an unpredictable 50/50 branch $m\to 0.5$, giving expected cost per iteration

$$\mathbb{E}[t_{\text{branch}}]=m\cdot r\approx 0.5\times 15=7.5\text{cycles}.$$

The mechanism (§3) is that $m$ depends on the data order, not the data: sorted runs make the branch predictable ($m\to 0$), random order does not. A branchless rewrite (arithmetic/conditional move, no control flow) removes $m$ entirely, costing $\sim 1$ cycle unconditionally.

2.4 Why UB is a performance bug, not a footnote

The compiler is entitled to assume no signed overflow, no strict-aliasing violation, no data race. It optimises using those assumptions. So UB does not “produce garbage at runtime”; it lets the compiler eliminate or reorder the very code you benchmarked, so your numbers describe a program that no longer exists. A data race in a hot loop can turn a “1 ns” measurement into a fiction. Correctness and performance are entangled: fix the UB and the benchmark becomes meaningful.

2.5 The cost hierarchy, ranked (first principles)

Sorting the failure modes by the latency each injects:

$$t_{\text{DRAM miss}}\sim 250c\gg t_{\text{allocator (slow path)}}\sim \text{100s–1000s ns}\gg t_{\text{mispredict}}\sim 15c\gg t_{\text{L1 hit}}\sim 4c.$$

Optimise top-down: eliminate misses and allocations first; branches and arithmetic are last.

---

3. Computational Implementation - two failures, modelled

Experiment 1 - the vector growth model (§2.1). Simulate doubling growth vs reserve(), counting reallocations and element copies, and expose the spike.

Reading Experiment 1. Growing to $10^6$ elements costs 20 reallocations and 1,048,575 element copies - about 1.05 copies per element (the $C(N)\approx N$ law of §2.1) with a single worst spike of 524,288 copies (~12 MiB memcpy) landing at an arbitrary tick. reserve(N) reduces all three to zero. This is exactly the “amortised but unbounded” trap: the average is fine, the spike destroys a latency budget.

Experiment 2 - branch misprediction (§2.3). Model a 2-bit saturating predictor over random vs sorted data; the same comparison costs 7.5 cycles/iter unsorted and ~0 sorted.

Reading Experiment 2. The comparison is identical; only the order differs. Random data defeats the predictor (50% miss → 7.5 cycles/iter of pure flushed pipeline), while runs on sorted data make it nearly free. The lesson is structural: either make the data predictable (sort/bucket before the hot loop) or make the code branchless. In trading, the usual answer is branchless, because you cannot reorder the market’s arrivals.

Practice checklist (from the failures). Pre-allocate with reserve/pools; keep the hot path free of new/throw/virtual/std::function; use SoA layouts; write branchless hot loops; run -fsanitize=address,undefined and -fsanitize=thread in CI to catch UB and races before they corrupt the benchmark; and never trust a measurement taken without a profiler (see 06 · Advanced Extensions).

---

4. Failure Modes & First-Principles Breakdowns (numbered)

1. Hidden allocation → tail explosion (§2.1, §3 Exp 1). The failure is amortised but unbounded: a push_back past capacity, a make_shared, a std::function box. Mean latency barely moves; p99.9 spikes. Cause: allocator lock/kernel fault. Fix: reserve, pools, value types.
2. Cache misses → latency-bound loop (§2.2). A locality-blind AoS layout fetches $3\times$ the needed bytes; each miss is ~250 cycles the prefetcher may not hide. Cause: layout (and pointer-chasing containers). Fix: SoA/flat containers, sequential access.
3. Undefined behaviour → a benchmark that lies (§2.4). Signed overflow, strict-aliasing, and data races let the optimiser delete/reorder the measured code. Cause: C++‘s “no UB” optimiser assumption. Fix: sanitizers, std::atomic, strict-aliasing-safe access.
4. Branch misprediction → pipeline flushes (§2.3, §3 Exp 2). Unpredictable control flow costs $m\cdot r$ cycles; data order alone changes it by orders of magnitude. Cause: speculation + unpredictable condition. Fix: branchless/cmov, sort/bucket inputs.
5. The micro-optimisation fallacy (§2.5). Hand-tuning arithmetic while the cost is at the memory or allocator tier. Cause: optimising the wrong level of the hierarchy. Fix: profile first, measure percentiles, optimise top-down.
6. False sharing across threads. Per-thread state sharing a cache line serialises on coherence traffic - a latency bug that looks like no bug. Fix: alignas(64) (§03). Cross-listed to Concurrency & Lockless Programming.

---

5. References

• Meyers, Scott: Effective Modern C++
• Fog, Agner: Optimizing Software in C++
• Bryant & O’Hallaron: Computer Systems: A Programmer’s Perspective
• Williams, Anthony: C++ Concurrency in Action (2nd ed.)
• Ghosh, Sourav: Building Low Latency Applications with C++

---

6. Connected Graph Bridges

• Back: 04 · Zero-Cost Abstraction · Index Hub
• Continue: 06 · Advanced Extensions
• Sibling: Concurrency & Lockless Programming (races, false sharing, memory fences)
• Sibling: Production Risk Guards & Kill Switches (pre-trade checks must not reintroduce hidden work)
• Base: Numerical Methods (floating-point UB-adjacent pitfalls and precision)