2.7.3 Colocation, Fiber, Microwave & the Speed-of-Light Limit

Basic Prerequisites: 01 · From Zero (units and the speed-of-light anchors).

---

1. Intuition & Practical Objective

This page is the physical layer of Pillar 2: where the matching engine sits, how close you must be to it, and what transport (fiber vs microwave) carries your signals. The objective is to make the design decisions concrete - how much latency each choice buys or costs, down to the microsecond.

The three inputs that decide the race:

1. Colocation distance. Racks inside the exchange’s data centre put your transmission into the nanosecond budget (tens of ns of NIC/DMA/parse). Being a few km off-site costs ~5 µs/km on fiber - an order of magnitude more than the entire colocated software stack. O’Hara (2015) gives the clean illustration from the Tokyo Stock Exchange: the standard Arrownet route is “several milliseconds”; the priority route ~260 µs; colocating compute at the primary site drops it to 15.7 µs.
2. Propagation medium. Glass fiber is $0.67c$ (≈5.00 µs/km). A microwave link in air is $0.9997c$ (≈3.34 µs/km). On straight paths microwave buys a ~33% cut in pure propagation - which is exactly why microwave replaced the “obsolete” $300M Spread Networks fiber cable within two years (BCS §1: fiber 13 ms → microwave 10 → 9 → 8.5 ms RT for NY↔Chicago).
3. The physics floor. Microwave cannot beat $d/c$. For London–Frankfurt the great-circle floor is 4.37 ms round-trip; Perseus’s microwave network achieves <4.6 ms - i.e. within ~5% of a physical constant. For London–New York the fiber floor is ~37 ms RT; the Hibernian undersea cable gets 59.6 ms (O’Hara 2015).

The one-sentence essence. “Colocation collapses the propagation term to the nanosecond scale; beyond it, transport is a physics problem where straight-air beats bent-glass by ~33% and where the realized latency approaches the speed-of-light floor to within a few percent - so the only remaining ‘speed’ is whoever built their geometry best, not whoever wrote better code.”

---

2. Mathematical Ground Truth & Derivations

Per-km latency. With refractive index $n$:

$$v=\frac{c}{n},L_{\text{per km}}=\frac{1000}{v}\mu \text{s}\Rightarrow \{\begin{array}{l}\text{fiber }(n\approx 1.5):5.00\mu \text{s/km},\\ \text{air: }3.34\mu \text{s/km}.\end{array}$$

Two-hop latency with displacement. If a rival is located $d$ km farther from the engine, you win by $\Delta t=d\cdot L_{\text{per km}}$. Being 1 km closer ≈ 5.0 µs (fiber) or 3.3 µs (air) - bigger than a typical colocated T2T budget (≈3.5 µs from the hub).

Round-trip time (RTT) for a transport of length $d$:

$$\text{RTT}(d)=2\frac{d}{v}.$$

The verified anchors (from hub §2, recomputed in §3):

Path	great-circle $d$	fiber RT (0.67c)	air RT (0.9997c)	verified realized
NY↔Chicago	1179 km	11.80 ms	7.87 ms	13 ms fiber (BCS) · 8.5 ms µwave (BCS)
London↔Frankfurt	655 km	6.56 ms	4.37 ms	<4.6 ms µwave (O’Hara)
London↔New York	5570 km	55.7 ms	37.2 ms	59.6 ms fiber (O’Hara)

Note London–NY: the realized 59.6 ms exceeds even the straight-fiber ideal 55.7 ms because an undersea fiber route is longer and includes optical/electrical regeneration - yet it is the same order as physics demands, underscoring that the race is now about geometry, not code.

Why the geometry wins. $d$ in the great-circle formula is straight-line distance; fiber hugs railroad/terrain curves, microwave flies straight over horizon towers. The ratio of what you can buy per kilometre, air vs fiber, is $0.9997/0.6666=1.50$ - i.e. microwave is 50% faster in speed, which is ~33% less latency per km ($1-0.6666/0.9997$) on an air path, before line-of-sight constraints.

---

3. Computational Implementation - colocation physics in numbers

Runs on the standard library only. It turns the §2 anchors into a live “you are here - why that costs you microseconds” model.

The 2 km line is the practical takeaway: a 2 km tape displacement costs 10 µs on fiber - roughly three times the entire colocated software T2T budget. That single fact is why nobody serious runs latency-critical racing from off the exchange campus.

---

4. Failure Modes & First-Principles Breakdowns

1. Geometric overpromise. Microwave’s 33% edge assumes a straight line of sight; real paths take tower zig-zags and weather, so realized gains shrink. Always recompute realized RT ($2d_{\text{actual}}/v$) with the real route length, not great-circle.
2. Under-budgeting the media. Using $c$ instead of $0.67c$ for fiber understates propagation by 50%. Pathologies: a colo decision at 5 km looks “free” but is a 25 µs handicap before routing.
3. Treating the floor as the design. Achieving 4.6 ms vs a 4.37 ms floor looks great, but the residual 0.23 ms is where switches, regeneration, and queueing hide - and it is exactly what a rival co-located closer eliminates. The margins, not the headline ms, decide the race (see 05 · Failure Modes).
4. Ignoring the matching-engine queue. Propagation is only one term; a stale-queue backlog at the engine can add milliseconds that dwarf your 5 µs distance edge - the full budget $T_{\text{prop}}+T_{\text{nw}}+T_{\text{mach}}$ always governs.

---

5. References

• O’Hara, Maureen (2015) - “High-Frequency Market Microstructure,” J. Financial Economics
• Budish, Cramton & Shim (2015)
• MacKenzie, Donald (2021) - Trading at the Speed of Light
• Menkveld (2013)

---

6. Connected Graph Bridges

• Back: 02 · The Latency Race · Index Hub
• Forward: 04 · Clock Synchronization (once you’re moved, you must agree on time)
• The software that runs inside the budget: Low-Latency Systems Architecture · High-Performance C++ for Trading · Hardware Acceleration & FPGA
• Exchange connectivity: FIX Protocol & Exchange Connectivity