8.4.1 Low-Latency Linux & Networking from Zero

Basic Prerequisites: Low-Latency Systems Architecture (the one-pillar systems frame). No kernel or networking background assumed.

---

1. Intuition & Practical Objective

This page builds the why of low-latency Linux & networking with no prior systems knowledge needed. The objective is one idea: a trading host is not a general-purpose computer - it is a machine with one job (turn a market-data packet into an order as fast and as predictably as possible), and the OS’s default behaviour actively works against that job.

Start with the dumbest question: why is latency a problem at all? A computer’s CPU is staggeringly fast - a 3 GHz core executes a billion instructions a second. But the data it needs is elsewhere: on the wire, in the NIC, in kernel memory, in a buffer you have to copy out of. Every hop from the wire to your application code is a transfer of ownership that costs time, and Linux has layered safety and fairness on top of every hop. The result: a packet takes ~10–25 µs and multiple microseconds of randomness (jitter) to cross the OS.

Three steps:

1. A fast mean is not fast. When two engines race to trade the same quote, the loser is decided by the tail - the rare slow packet - not the average. One 100 µs stall loses the trade even if the other 99,999 packets arrived in 8 µs. So the discipline is about removing variance, not just lowering the mean.
2. The kernel is doing the wrong thing for you. Linux tuned the network path for throughput and fairness: it interrupts the CPU on packet arrival, switches into kernel mode, copies the packet, runs protocol layers, copies again, and wakes your thread. Every one of those steps is a context that costs microseconds and injects randomness. A low-latency host deletes the ones it can.
3. Determinism is a budget decision, not an accident. A pinned core that busy-polls the NIC (spins, never sleeps) burns 100% CPU doing “nothing” between packets. That looks wasteful - but it is deliberately trading a cheap, schedulable resource (CPU time) for the expensive one (deterministic latency).

The one-sentence essence. “Between the wire and your strategy code sits an OS that was built to be fair, safe, and high-throughput - and a low-latency host’s whole job is to get it out of the way, one interrupt, copy, and scheduler wake at a time, then prove the result with honest measurement.”

---

2. Mathematical Ground Truth & Derivations

The additive latency model. The tick-to-market-data path is a chain of stages; total latency is their sum:

$$T_{\text{RX}}=T_{\text{wire}}+T_{\text{NIC}}+T_{\text{stack}}+T_{\text{parse}}.$$

Because it is a sum, the slowest stage dominates the mean. The median packet is the “usual” one; the packet that decides a race is the one at the 99th percentile, and it is governed by the stage with the largest variance, not the largest mean.

Why the tail is where you lose. For a latency random variable $X$, the p99.9 quantile $q_{99.9}$ satisfies $\mathrm{Pr}[X\le q_{99.9}]=0.999$. When stage latencies are independent,

$$\mathrm{Var}(T)=\sum _i\mathrm{Var}(X_i),$$

so a single high-variance stage sets every upper percentile. Concretely (verified in §3): an end-to-end path whose median is ~8 µs but whose stack occasionally spikes to ~45 µs has p99 ≈ 53 µs and max ≈ 289 µs - the 2% of packets that hit the spike own the tail.

How fast can it possibly be? Physics bounds the wire: light travels ~20 cm/ns in fibre, so ~5 ns per metre one-way. No software makes a 1 200 km fibre path faster than ~6 ms - that is why colocation, not code, is the first lever (see 03 · Colocation & Networks).

---

3. Computational Implementation - the latency distribution & the tail

The most convincing way to see why the tail matters: simulate a market-data path with and without a rare interrupt/scheduler spike, then read off the percentiles. Stdlib only.

The median barely moves (8.01 → 8.08 µs) and the mean grows by only ~1 µs, but the p99 explodes from 17.9 to 52.9 µs and the max from 37.6 to 288.9 µs - driven by just 2% of packets. That is the whole lesson of this folder: a system that looks fine on its average is losing races on its tail, and the tail is created by a few rare events you must engineer away.

---

4. Failure Modes & First-Principles Breakdowns

1. “The average is good, why is it slow?” You quoted the mean; the market charges the tail. Report min and percentiles, never mean (this is the seed of the measurement discipline in 06 · Advanced Extensions).
2. Rare events are the enemy, not slow events. A 2% spike (an interrupt storm, a scheduler wake, a TLB miss) owns the p99. Engineering determinism - pinning, busy-polling, hugepages - is how you delete the 2%, and it is worth more than shaving the other 98%.
3. “The OS is free.” It is not - every copy, context switch, and fairness preemption is a transfer of ownership with a latency and a jitter cost. Counting those transfers (the systems view in 02 · Latency Hierarchy) is the first step.

---

5. References

• Kerrisk, Michael - The Linux Programming Interface (No Starch, 2010). The syscall/socket/timer reference behind everything a tuning page says.
• Stevens, W. Richard (Fall, Kevin) - TCP/IP Illustrated, Vol. 1, 2nd ed. The transport semantics.
• lowlatencysystem.com - The Complete Guide to Low-Latency Trading Systems. The gentle, layered orientation read: tick-to-trade, OS/Linux tuning, network path.
• Databento - Low-Latency Tuning Guide for Linux and Trading Systems. Practitioner-grade; the next step after this page.

---

6. Connected Graph Bridges

• Forward: 02 · Kernel Tuning · Index Hub
• Systems frame: 02 · Latency Hierarchy · Low-Latency Systems Architecture
• Physics/colocation: 03 · Colocation & Networks
• Statistics: Probability & Measure Theory (quantiles, variance decomposition)