2.9.3 The HFT System Architecture

Basic Prerequisites: 02 · The Latency Hierarchy.

---

1. Intuition & Practical Objective

Almost every latency-sensitive trading system, from a two-person prop shop to a top-tier HFT, is the same three-stage pipeline:

$$\underset{\text{decode market data}}{\underbrace{\text{Feed Handler}}}⟶\underset{\text{decide}}{\underbrace{\text{Strategy}}}⟶\underset{\text{encode \& send orders}}{\underbrace{\text{Order Gateway}}}.$$

The stages are connected by one-way message channels (in practice, ring buffers - 04). The objective of this page is to understand the architecture as a queueing system, because that is the only rigorous way to answer the questions that actually matter:

• How loaded should each stage be? (Answer: far below 100 % - §2.)
• Where does jitter come from? (Answer: queueing delay, which grows without bound as load approaches 1.)
• What does “back-pressure” cost? (Answer: if the fast stage waits on the slow stage, the fast stage’s latency becomes the slow stage’s latency.)

The guiding principle throughout is one thread, one job, one direction of data flow - a design in which each stage is a simple, cache-resident loop that never blocks on a lock and never allocates. This is the same “do less work” principle from 01, now applied to how the stages are wired together.

The one-sentence essence. “A trading engine is a chain of single-writer queues; its latency is the sum of stage service times plus queueing delay, and queueing delay explodes as each stage’s utilisation approaches 1 - so the architecture must keep every stage comfortably idle, not busy.”

---

2. Mathematical Ground Truth & Derivations

2.1 The stages and their messages

• Feed handler. Receives the venue’s multicast feed (e.g. a binary ITCH-style protocol, or FIX for some venues - see FIX Protocol & Exchange Connectivity), decodes it into a normalised tick, and publishes it. It is a producer: it never waits for the strategy.
• Strategy. Consumes ticks, maintains whatever view it needs (book, position, signals), and emits orders/cancels. It is the only place the trading decision lives.
• Order gateway. Encodes orders into the venue’s wire format, manages session/sequence state, and writes them to the socket. It is the last stage and must never stall the strategy.

Data flows one way. Market data flows downstream; there is no call from the strategy “up” into the feed handler and no blocking handshake between stages. Blocking is what turns two independent latencies into one correlated disaster.

2.2 Queueing model: load sets latency

Treat each stage as a single server. Arrivals are message events; the service time is the stage’s processing cost. The canonical results (Kendall notation $M/M/1$, i.e. Poisson arrivals, exponential service, one server):

$$L=\lambda W(\text{Little’s law}),W_q=\frac{\rho }{1-\rho }\mathbb{E}[S],\rho =\lambda \mathbb{E}[S].$$

The mean time in system is $W=W_q+\mathbb{E}[S]=\frac{\mathbb{E}[S]}{1-\rho }$. At $\rho =0.5$, $W=2\mathbb{E}[S]$; at $\rho =0.9$, $W=10\mathbb{E}[S]$; at $\rho =0.99$, $W=100\mathbb{E}[S]$. A stage that looks “twice as fast as needed” is not wasting half its capacity - it is buying a 5x reduction in mean sojourn time (a 9x cut in queueing delay proper, $W_q$).

For general service-time variability the Pollaczek–Khinchine formula generalises this:

$$W_q=\rho \mathbb{E}[S]\frac{1+C_s^2}{2(1-\rho )},C_s=\frac{\mathrm{std}(S)}{\mathbb{E}[S]}.$$

Two design laws follow directly:

1. Keep $\rho$ low (the folklore “run hot - never above ~50–70 %”): queueing delay is a reciprocal function of the margin to saturation.
2. Keep $C_s$ small - deterministic service (constant-time hot loop, no allocation, no branch surprises) is worth as much as raw speed, because jitter multiplies the queueing term.

When arrivals are deterministic and service is deterministic ($D/D/1$), there is no randomness and delays stay bounded; but real feeds are bursty (a batch of ticks arrives together, e.g. after an auction), so the realistic model is closer to $M/D/1$ or a bursty/GI/D/1 queue - which is exactly why a ring buffer with headroom matters: it absorbs bursts without forcing the strategy to wait.

2.3 The design principles (compressed)

These are the recurring rules of the discipline; each is a direct consequence of §2.1 of the previous page:

• Do less work. Latency $\approx$ instructions executed on the critical path; every branch and copy costs real cycles.
• No allocation on the hot path. malloc/new/GC introduce 50–500+ ns heap or lock costs and unbounded tail pauses. Pre-allocate everything at startup (arenas).
• Mechanical sympathy. Design for the hardware: contiguous memory, streaming access, cache-line-aligned (64 B) fields, branch-predicted common cases.
• Single writer per queue. One producer and one consumer per channel removes the need for locks entirely (04).

---

3. Computational Implementation - the pipeline as a queue

Standard library only. We simulate the strategy stage as a single server with Poisson arrivals and log-normal service times, sweeping the offered load $\rho$, and measure the resulting queueing delay, sojourn time, and the p99. The Little’s-law column ($L=\lambda W$) is computed from the actual time-in-system integral, so it doubles as a check on the simulation.

Read the result. Raw service time is constant at 1 µs, yet the observed sojourn time goes 1.2 → 1.7 → 4.0 → 111 µs as load rises through 20 % → 95 %. The stage did no extra work - the latency is entirely queueing delay. The p99 at $\rho =0.95$ (351 µs) is 15 000x the 20 %-load p50: a “busy” engine is a latency bomb, which is why production systems cap stage utilisation far below saturation even at the cost of idle hardware. The Little’s-law check ($L\approx 3.200$ vs $\lambda W=3.199$) confirms the simulation conserves work.

---

4. Failure Modes & First-Principles Breakdowns

1. Running each stage hot. Because $W_q=\rho \mathbb{E}[S]/(1-\rho )$, the marginal cost of load is nonlinear: the last 10 % of utilisation buys ten times the delay. Capacity planning is latency planning.
2. Blocking between stages. If the strategy blocks waiting on the feed handler (a lock, a synchronous call), the two latencies become correlated and their tails add; the single-writer one-way design exists to forbid exactly this.
3. Unbounded queues. A ring buffer that can grow without limit converts a latent overload into an ever-growing latency instead of an explicit drop - you lose the tick’s staleness signal. Bounded ring + explicit drop policy is the honest design (04).
4. Jittery service time. $C_s$ in the P-K formula multiplies the queueing delay; a stage with occasional 100 µs pauses queues like a stage with a much higher mean. Determinism is a first-class requirement, handled in 05 · Failure Modes.

---

5. References

• Fowler, Martin - The LMAX Architecture and the LMAX Disruptor paper. The canonical one-way, single-writer, ring-buffer architecture this page describes.
• NautilusTrader documentation (nautilustrader.io). A production-grade, open-source event-driven engine whose message-bus/actor design is the cleanest modern reference for stage decoupling.
• Kerrisk, Michael - The Linux Programming Interface (2010), and Benvenuti - Understanding Linux Network Internals (2
• Gregg, Brendan - Systems Performance (2nd ed., 2020). Measuring per-stage latency and queueing in production.
• FIX Protocol official specifications (fixtrading.org) and Nasdaq TotalView-ITCH 5.0 spec. The actual wire protocols the feed handler and gateway speak.
• Hamilton, Peter et al.

---

6. Connected Graph Bridges

• Back: 02 · The Latency Hierarchy
• Forward: 04 · Lock-Free & Ring Buffers · Index Hub
• Cross-pillar: FIX Protocol & Exchange Connectivity · Concurrency & Lockless Programming
• Deeper: Hardware Acceleration & FPGA (move the feed handler into silicon)