8.2.6 Advanced Extensions

Basic Prerequisites: 05 · Failure Modes and 03 · Memory & Cache.

---

1. Intuition & Practical Objective

You cannot optimise what you cannot measure, and in low-latency work the obvious measurement is wrong. Naïve benchmarking - wrap a timer around one call, average a handful of runs - reports timer overhead, scheduler noise, and warmup effects rather than the work. This page is the launchpad for the practice: how to benchmark honestly, how to profile to find the real cost, and how to configure the toolchain (compiler, flags, link-time optimisation, profile-guided optimisation, sanitizers) so the binary you ship is the binary you measured.

Three “aha”s:

1. Timing one call measures the clock. A perf_counter/clock_gettime call itself costs tens of nanoseconds - the same order as the operation. Batching many repetitions inside one timed region and dividing is the only honest micro-benchmark (§3).
2. Mean is the wrong statistic. Report min (least noise) and percentiles (the tail is what the market charges). A single scheduler spike of 12 µs corrupts any mean.
3. The toolchain is part of the algorithm. -O2 vs -O3, -march=native, LTO, and PGO routinely move latency by 10–50%; perf/VTune/heaptrack turn “it feels slow” into a named hot line. Ship with the flags you measured and lock them in the build.

One-sentence essence. “Measure with percentiles after warmup, in batches; profile to find the true cost; then freeze the exact compiler flags and tooling so tomorrow’s build performs like today’s measurement.”

---

2. Mathematical Ground Truth & Derivations

2.1 Estimator bias: why single-shot timing overshoots

Let the true per-op time be $\mu$ and let the timer/measurement overhead be $ϵ$ (the cost of reading the clock around the operation). A single-shot estimate is

$$\hat{t}=\mu +ϵ,\frac{\hat{t}}{\mu }=1+\frac{ϵ}{\mu }.$$

When $\mu \sim 20$ ns and $ϵ\sim 100$ ns, the estimator is inflated by $\approx 6\times$ - you measured the clock, not the work. Batching $B$ operations inside one timed region amortises the overhead:

$${\hat{t}}_B=\mu +\frac{ϵ}{B}\underset{B\to \infty }{\to }\mu .$$

The bias falls as $1/B$; with $B=10^5$ the overhead term is negligible. §3 measures exactly this: batching shrinks the estimate - proving the single-shot number was mostly clock.

2.2 Percentiles vs the mean

For a latency sample, the pth percentile $t_p$ satisfies $F(t_p)=p$, while the mean integrates the whole tail:

$$\mathbb{E}[T]={\int }_0^{\infty }(1-F(t))dt.$$

Because $1-F(t)$ is tiny in the tail but the values are huge, the mean under-reports the tail by construction. Rule: always report $\min$ (the noise floor / best achievable) and $t_{99},t_{99.9}$ (what the strategy actually experiences). The mean is reported only to compare against the median.

2.3 Variance and the noise floor

Repeated timings $t^{(1)},\dots ,t^{(K)}$ have sample variance $s^2=\frac{1}{K-1}\sum (t^{(i)}-\bar{t}{)}^2$. Scheduler preemption, frequency scaling, and SMT contention inject outliers that inflate $s^2$ without touching $\min$. Hence:

$$\underset{k}{\min }{t}^{(k)}\approx \text{signal (least noise)},\bar{t}\approx \text{signal}+\text{noise}.$$

A stable micro-benchmark is one where $\min$ is reproducible to a few percent across runs; if $\min$ wanders, the machine (governor, turbo, other tenants) is not under control and no conclusion is valid.

2.4 Speedup accounting

For an optimisation changing a stage’s time from $t$ to $t^{\prime }$, Amdahl’s law bounds the end-to-end gain by the stage’s share:

$$S_{\text{total}}=\frac{1}{(1-p)+\frac{p}{S_{\text{stage}}}},$$

where $p$ is the fraction of total time in the optimised stage. Optimising a 5%-share stage by $2\times$ yields only $\approx 1.025\times$ end-to-end - which is why profiling (finding $p$ large) must precede optimisation (05 · Failure Modes, §2.5).

2.5 The compiler’s optimiser as a search over equivalences

Optimisation levels enable transformations (inlining, unrolling, vectorisation, loop-invariant hoisting) whose payoff is data-dependent - -O3/-march=native can be slower on some loops (code bloat, instruction-cache pressure, worse register allocation). LTO lets those transformations cross translation-unit boundaries; PGO feeds the compiler real branch/loop profiles so it optimises the actual hot path. The engineering rule: choose flags by measurement on the target workload, not by folklore, and pin the winning set in the build.

---

3. Computational Implementation - benchmark methodology, demonstrated

Standard library only. The example times a tiny operation both ways - single-shot (biased) and batched (honest) - and prints the sample spread that justifies percentile reporting. Numbers are machine-dependent; the ratios are the lesson.

Reading the result. Single-shot timing reports a mean of 171.3 ns against a batched 103.2 ns/op - the single-shot estimator is inflated by 1.7× because it includes the clock read (and the numbers would be far worse for a truly 5 ns operation). The spread line is the other half of the lesson: the samples range from 160 ns to 12,678 ns, an ~79× outlier from a scheduler spike. A lone mean is a lie; min + percentiles is the honest summary. (In C++ the same discipline uses std::chrono::steady_clock, batch loops, Google Benchmark’s DoNotOptimize/ClobberMemory to stop the optimiser deleting the work, and reports cpu_time + percentiles.)

Toolchain & practice (prose).

• Compile flags: -O2/-O3 baseline; -march=native (or an explicit -mavx2/-mavx512f) for SIMD; -flto for cross-TU inlining; PGO (-fprofile-generate → run → -fprofile-use) for real branch/loop data; -DNDEBUG to strip asserts from the hot path (keep them in the test build).
• Profiling: perf stat (IPC, cache-misses, branch-misses), perf record + flamegraphs (find the hot line), perf c2c (false sharing / cache-line contention), Intel VTune (microarchitecture), heaptrack/valgrind --tool=massif (allocations - cross-check §05).
• Correctness gates: -fsanitize=address,undefined and -fsanitize=thread in CI (they make the benchmark trustworthy - §05, §2.4 of 05); lock the build with CMake presets so every engineer measures the same binary.
• Benchmark harness: Google Benchmark (DoNotOptimize, ClobberMemory, statistical repeats) for micro-benchmarks; run under taskset/chrt with the performance governor and turbo pinned (Low-Latency Systems Architecture).

---

4. Failure Modes & First-Principles Breakdowns

1. Dead-code elimination in the benchmark. If the result is unused the optimiser deletes the work and reports “0 ns”. Fix: DoNotOptimize/ClobberMemory, consume the output, or benchmark a checksum.
2. Timer overhead / insufficient repetitions (§2.1, §3): a 1.7× inflation here, worse for smaller ops. Fix: batch and divide; prefer hardware counters (rdtsc, perf) for sub-100 ns work.
3. No warmup. First iterations pay page faults, instruction-cache fills, branch-predictor cold starts, and (in managed layers) JIT. Discard warmup, then measure the steady state.
4. Statistics abuse (§2.2–2.3): a lone mean, or comparing runs on an uncontrolled machine (turbo, governor, other tenants). Fix: report min + p99/p99.9, pin the environment, repeat across runs.
5. Optimising the wrong stage (§2.4): Amdahl caps the gain at that stage’s share. Profile before optimising; the top line in perf is rarely where you guessed.
6. Flags that don’t match the target. -O3 -march=native on the build box but deployed on a different microarchitecture → illegal instruction or lost vectorisation. Pin flags to the deployment target and re-benchmark.
7. UB corrupts the benchmark itself (§05, §2.4): a data race or aliasing violation lets the optimiser transform the measured code. Sanitizers before speed claims.

---

5. References

• Fog, Agner: Optimizing Software in C++
• Bryant & O’Hallaron: Computer Systems: A Programmer’s Perspective
• Ghosh, Sourav: Building Low Latency Applications with C++
• Gregg, Brendan: Systems Performance (and BPF Performance Tools)
• Williams, Anthony: C++ Concurrency in Action (2nd ed.)

---

6. Connected Graph Bridges

• Back: 05 · Failure Modes · Index Hub
• Forward: Concurrency & Lockless Programming (TSan, contention, perf c2c) · Hardware Acceleration & FPGA (when the CPU toolchain is not enough)
• Sibling: Low-Latency Systems Architecture (governor, taskset, kernel bypass - the measurement environment)
• Base: Pillar 8: Quantitative Development