2.9.5 Failure Modes & Practice

Basic Prerequisites: 04 · Lock-Free & Ring Buffers.

---

1. Intuition & Practical Objective

Low-latency systems are not defeated by being slow; they are defeated by being unpredictable. A path that takes 2 µs every time wins races against a path that takes 1 µs on average but 50 µs once in a thousand ticks, because the race is decided on the tick that matters, and that tick is by construction a laggard. This page names the failure modes precisely, so a practitioner knows which assumption to distrust and how each failure shows up in nanoseconds. The objective is the discipline of knowing where your tail comes from and eliminating it at the source rather than shaving the median.

The recurring culprits, in one line each:

1. Allocation / garbage collection in the hot path - a stop-the-world event measured in microseconds lands inside a nanosecond-scale path.
2. Cache misses and false sharing - touching data the CPU doesn’t have, or fighting another core for a cache line.
3. The OS getting in the way - timer interrupts, context switches, page faults, syscalls.
4. Load-induced queueing - the engine is fine but its utilization is too high (03).

The one-sentence essence. “The median is vanity; the p99.9 is sanity - and the p99.9 is set by stop-the-world events (allocation, page faults, scheduler preemption, cache misses), not by the average instruction count, so the fix is to remove the events, not to shorten the average.”

---

2. Mathematical Ground Truth & Derivations

2.1 The tail is set by the worst event, not the average

Let a hot path have a baseline cost and, with small probability $p$, a “hiccup” of size $H$ (a GC pause, a page fault, a scheduler preemption). The quantile at tail probability $p$ is

$$Q_{1-p}\approx \text{baseline}+H,$$

so the p99.9 - the statistic that decides close races - is baseline $+H$ whenever $p\gtrsim 10^{-3}$. Reducing the baseline does nothing to the tail; removing the hiccup removes the tail in one step. This is why “profile the mean, optimize the mean” is the wrong loop for HFT.

2.2 Why allocation is catastrophic

Every dynamic allocation can trigger: a heap free-list search, a possible lock, and (periodically) a page fault when the heap grows. GC-based runtimes add scan-and-stop pauses proportional to the live set. The costs land on a spectrum:

Event	Cost	Frequency	Dominates
Free-list malloc	~50–500 ns	per call	median
Page fault (heap growth)	~1–10 µs	per new page	p99.9
GC pause (managed runtime)	~0.1–10 ms	periodic	p99.99

The cure is structural, not an optimization: pre-allocate a fixed arena at startup, and never allocate inside the event loop. This is the same rule as “fixed ring buffer” from 04 applied to memory in general.

2.3 Why cache misses and false sharing break the model

The memory hierarchy (02) makes where data lives part of the algorithm:

• Hit rate. If a structure of size $S$ exceeds the cache of size $K$, misses happen with probability roughly $1-K/S$ under random access; each miss costs 60–100 ns. Latency $\approx$ (hit rate)$\times$1 ns $+$ (miss rate)$\times$80 ns.
• False sharing. Two cores writing to independent variables that share one 64-byte cache line cause the MESI protocol to bounce that line between their caches on every write; a ~1 ns store becomes a ~100–1000 ns coherence round trip. The fix is padding to 64-byte boundaries.
• NUMA. On multi-socket hardware, memory attached to the other socket costs +40–80 ns per access, silently, on every load. Fix by pinning threads and allocating memory on the socket that owns the NIC.

2.4 The OS as a latency source

Standard Linux is designed for throughput and fairness, not latency determinism; the scheduler will preempt your thread for timer ticks, background work, and interrupts. A context switch costs 1–3 µs and pollutes the cache; a page fault can cost microseconds; a syscall costs hundreds of nanoseconds and can sleep. This is the root cause of the fat receive-path tail measured in the hub. The architectural responses - isolcpus, nohz_full, IRQ affinity, mlockall, huge pages, busy-polling, kernel bypass - are the subject of 06 · Advanced Extensions.

---

3. Computational Implementation - seeing the tail fail

Standard library only. Two experiments: (a) tail amplification by composition across six independent hops, and (b) a stop-the-world hiccup injected into a nanosecond-scale path, showing the median is untouched while the p99.9 explodes.

Read the result. (a) Each hop is in its own worst 0.1 % with probability 0.001, yet at least one is in the tail 0.52–0.60 % of the time - the ~6x amplification predicted by $1-(1-q{)}^k$. Per-stage “rare” events are system-level “routine” events. (b) Injecting a hiccup in just 0.1 % of messages leaves the p50 (2 003 ns) and p99 (3 625 ns) untouched, but moves the p99.9 from 4 311 ns to 51 499 ns (a 50 µs pause) or 501 499 ns (a 500 µs pause) - the full size of the pause, exactly as §2.1 predicts. A single stop-the-world event class defines the tail; nothing about the median reveals it.

---

4. Failure Modes & First-Principles Breakdowns (numbered)

1. Allocation on the hot path. malloc/new/GC in the event loop injects 50 ns–10 ms hiccups into a sub-microsecond path. The median is unchanged; the p99.9 is destroyed. Fix: pre-allocated arenas, no exceptions on the path, no std::string/std::map construction, no logging in the loop.
2. Cache misses on a cold structure. A “hot” table larger than L2 (or thrashed by a co-tenant thread) pays 60–100 ns per lookup. Fix: shrink the working set, make access sequential/streaming, prefetch.
3. False sharing. Independent counters sharing a cache line bounce between cores: a ~1 ns store becomes a ~100–1000 ns coherence round-trip. Fix: 64-byte alignment/padding of per-thread state.
4. NUMA-remote memory. Threads on socket 0 touching socket-1 memory pay +40–80 ns on every access. Fix: pin threads and allocate on the NIC’s node.
5. OS interference: interrupts, preemption, page faults, syscalls. Timer ticks and background work preempt the loop (1–3 µs), page faults cost microseconds, syscalls cost hundreds of ns and can sleep. Fix: isolcpus/nohz_full, IRQ affinity, mlockall, pre-faulted/touched memory, huge pages, busy-polling, kernel bypass.
6. High utilization. Even a perfectly deterministic stage queues catastrophically near $\rho =1$ (03). Fix: cap the load, add headroom, shed stale work explicitly.
7. Measuring the mean. A benchmark that reports only “average latency” cannot see any of the above. Fix: always report the distribution (p50/p99/p99.9/max), and measure under the production load and co-tenancy.

---

5. References

• Drepper, Ulrich - What Every Programmer Should Know About Memory (2007). Cache, false sharing, prefetch, NUMA
• Thompson, Martin - Mechanical Sympathy. Practical treatment of cache-line effects and false sharing in trading systems.
• Gregg, Brendan - Systems Performance (2nd ed., 2020). Methodology for finding the actual tail source (off-CPU analysis, flame graphs); and Gregg, BPF Performance Tools for latency histograms in production.
• Williams, Anthony - C++ Concurrency in Action (2nd ed., 2019). Memory model and false-sharing-avoidance patterns (cross-listed: Concurrency & Lockless Programming).
• MacKenzie, Donald - Trading at the Speed of Light (2021). How these engineering tails translate into who wins on the exchange.
• Hasbrouck & Saar - “Low-latency trading” (2

---

6. Connected Graph Bridges

• Back: 04 · Lock-Free & Ring Buffers · Index Hub
• Forward: 06 · Advanced Extensions
• Cross-pillar: Production Risk Guards & Kill Switches (what to do when the tail wins) · Concurrency & Lockless Programming
• Foundational: Numerical Methods (measuring distributions honestly)