2.9.2 The Latency Hierarchy

Basic Prerequisites: 01 · From Zero.

1. Intuition & Practical Objective

Every latency a trading system can experience is a point on a single hierarchy that spans nearly eight orders of magnitude, from ~1 ns (a register/L1 hit) to ~60 ms (a transatlantic round trip). The practical objective of this page is to internalise that hierarchy as a ruler, so that when you read ” $X$ nanoseconds” you immediately know which layer of the machine it belongs to and whether the number is plausible. The ruler also tells you what is fixable: a 100 ns cache miss is fixable by data layout; a 5 µs kernel-stack traversal disappears under kernel bypass; a 6 ms ocean crossing is only fixable by moving the box.

The mental model is a pyramid. At the tip: the CPU working on data it already has (registers, L1). Each step down is roughly an order of magnitude slower - L2, L3, DRAM, a device (NIC/disk), the kernel, another machine. The hot path of a trading engine is an argument for staying as close to the tip as possible, and the entire software design follows from that wish.

The one-sentence essence. “Latency is stratified: ~1 ns to touch data in L1, ~100 ns if it fell out to DRAM, ~10 000 ns if the kernel got involved, and milliseconds once light crosses an ocean - and you can only reduce which layer a step lands in, never the layer’s own cost.”

2. Mathematical Ground Truth & Derivations

2.1 The hierarchy (a ruler to memorise)

Layer	Latency	Cycles @ 3.5 GHz	Order of magnitude
Register / L1 hit	$\sim 1$ ns	$\sim 4$	$1 0^{0}$ ns
L2 hit	$\sim 3 - 4$ ns	$\sim 12$	$1 0^{0}$ ns
L3 hit	$\sim 10 - 15$ ns	$\sim 40$	$1 0^{1}$ ns
DRAM (local)	$\sim 60 - 100$ ns	$\sim 250$	$1 0^{2}$ ns
DRAM (remote NUMA)	$+ 40 - 80$ ns	-	$1 0^{2}$ ns
Mutex lock/unlock (uncontended)	$\sim 10 - 20$ ns	-	$1 0^{1}$ ns
`malloc`/`new`	$50 - 500 +$ ns	-	$1 0^{2}$ ns
System call	$\sim 100 - 1000$ ns	-	$1 0^{2} - 1 0^{3}$ ns
Context switch	$\sim 1000 - 3000$ ns	-	$1 0^{3}$ ns
Linux TCP/IP RX path	$\sim 5000 - 25000$ ns	-	$1 0^{4}$ ns
SSD read	$\sim 1 0^{5}$ ns (100 µs)	-	$1 0^{5}$ ns
Same metro round trip (colocated)	$\sim 1 0^{5}$ ns	-	$1 0^{5}$ ns
Ocean round trip (NJ–London)	$\sim 5.5 - 6 \times 1 0^{7}$ ns (~56 ms)	-	$6 \times 1 0^{7}$ ns

Read this as ratios, not absolutes. A DRAM access is ~100x an L1 hit. A kernel-stack traversal is ~15 000x an L1 hit. Those ratios are the only reason the architecture choices in this folder exist: each technique is a move from a lower row to a higher row.

2.2 The budget as a sum, and why the tail lives in one row

The tick-to-trade budget is additive (§hub). With log-normal stage costs $T_{i} \sim LN (ln m_{i}, σ_{i})$ , the total has a skewed distribution: even if every $σ_{i}$ is small, the sum inherits the largest $σ$ and its mean sits above its median. Formally, for the sum of independent log-normals no closed form exists, but the tail is governed by the heaviest stage:

Var (T) = i \sum Var (T_{i}), skew (T) \approx \frac{\sum _{i} skew ( T _{i} ) σ _{i}^{3}}{( \sum _{i} σ _{i}^{2} ) ^{3/2}} .

Practical rule: a fat-tailed stage is a tail-source for the entire path. In HFT the fat-tailed stage is almost always the kernel/network path ( $σ$ large, because a packet can wait behind a softirq, a scheduling decision, or a syscall). Removing it (kernel bypass) does not just lower the mean - it removes the skew.

2.3 The physics floor

Speed of light in fibre $\approx 20$ cm/ns $\Rightarrow\approx 5$ ns/m one way. Round trip $\approx 10$ ns/m plus switching. A colocated cross-connect (a few km) is $\sim$ tens of µs; continental is milliseconds. This term is a constant, and no software touches it - which is why the honest first decision is location, not algorithm.

3. Computational Implementation - decomposing the budget

Standard library only. We assign each stage of the tick-to-trade path a log-normal cost, sample 200 000 paths, and read off the mean/p50/p99/p99.9 both per stage and end-to-end, plus each stage’s share of the p99. This is the measurement exercise that precedes any optimization.

Read the result. The kernel/stack hop is one of six stages but owns 81.3 % of the end-to-end p99, and it is the stage that pushes the mean above the median and the p99 to ~2.6x the median. Every other stage is a rounding error at the tail. That is the ruler in action: the technique that removes the kernel/stack row (kernel bypass, §06/§hub) changes the whole distribution, while micro-optimising the 80 ns serialize stage cannot move the p99 measurably. Find the row that owns the tail before touching any code.

4. Failure Modes & First-Principles Breakdowns

Optimising the wrong row. Shaving 20 ns off the serializer when the kernel path contributes 4 800 ns to the p99 is invisible. Always attribute the tail by stage before optimizing.
Assuming “cache hit” without measuring. A structure that “should be small” can be far larger than L1 (32–64 KB) and miss on every message; a supposedly hot structure evicted by a neighbour thread is a 100 ns miss wearing a 1 ns disguise. (See 05 · Failure Modes.)
Ignoring NUMA placement. On a multi-socket box, a thread on socket 0 reading memory allocated on socket 1 pays the remote-DRAM premium on every access - a silent, systematic $\sim$ 40–80 ns tax on the whole hot path.
Forgetting the floor is physical. Colocation and route decide the millisecond-scale term; a strategy that needs sub-100 µs against a competitor two continents away is fighting physics, not code.

5. References

Drepper, Ulrich - What Every Programmer Should Know About Memory (Red Hat, 2007). The authoritative free treatment of the cache/memory numbers in
Thompson, Martin - Mechanical Sympathy (blog/talks). Cache lines, prefetch, and why predictable access patterns beat clever ones.
Gregg, Brendan - Systems Performance (2
Benvenuti, Christian - Understanding Linux Network Internals (2005) and Kerrisk, Michael - The Linux Programming Interface (2010). Where the kernel/stack microseconds actually come from.
Hasbrouck & Saar - “Low-latency trading,” J. Financial Markets 16(4) (2
MacKenzie, Donald - Trading at the Speed of Light (2

6. Connected Graph Bridges

Back: 01 · From Zero
Forward: 03 · System Architecture · Index Hub
Cross-pillar: High-Performance C++ for Trading (cache-locality implementation) · Numerical Methods (measuring distributions)
Sibling: Hardware Acceleration & FPGA (the sub-100 ns row)

Kwant Atlas

Explorer