8.9.6 Advanced Extensions

Basic Prerequisites: 05 · Failure Modes and Concurrency & Lockless Programming.

---

1. Intuition & Practical Objective

Everything so far assumed a single process that you could reason about. Real production systems run two or more copies for the obvious reason: a single machine, a single NIC, a single kernel, or a single power feed failing is a certainty over a trading year, and you cannot be down at the open. High availability is the engineering of staying in the market across those failures.

But redundancy introduces its own, sharper class of danger - and it is the danger that causes the truly spectacular losses:

• Split brain. Two copies both believe they are the primary and both trade, doubling positions and doubling risk. The classic disaster.
• Failover into a stale book. The standby takes over from a replicated state that is seconds old, so it starts with positions that do not match the venue and it does not know it (see 05 for why that is unrecoverable without reconciliation).
• Correlated failure. Both replicas run the same code with the same bug and fail together - the redundancy bought nothing, and you paid for twice the infrastructure. Redundancy only helps against independent failures.

The correct framing is a trade-off surface, not a binary “HA on/off”: you choose a target availability, and you pay for it in latency (synchronous replication), complexity (leader election, fencing), and cost. And you must be honest about your RTO (recovery time objective - how long you are out) and RPO (recovery point objective - how much state you can lose), because those two numbers, not “we have replicas,” are the actual specification.

The one-sentence essence. “Availability is a product of redundancy and independence: $A_N=1-(1-A{)}^N$ forgives independent replica failures but not common causes, so the engineering effort goes into (i) fencing to guarantee at most one primary, (ii) replication with a defined RPO, and (iii) a failover path whose RTO and false-failover rate you have actually measured - with reconciliation on reconnect as the non-negotiable last step.”

---

2. Mathematical Ground Truth & Derivations

2.1 Availability of a runtime and of $N$ independent replicas

With mean time between failures MTBF and mean time to repair/reprovision MTTR (same units),

$$A=\frac{\text{MTBF}}{\text{MTBF}+\text{MTTR}},\text{downtime per year}=(1-A)\cdot 8760\text{h}.$$

For $N$ replicas whose failures are statistically independent, availability is the parallel-redundancy product:

$$A_N=1-(1-A{)}^N.$$

The “nines” translation: $A=0.999$ is ~8.8 h/yr down; $A=0.9999$ is ~53 min/yr; $A=0.99999$ is ~5.3 min/yr.

2.2 The common-cause floor - the term that eats the model

Independence is the lie in $A_N$. Let a fraction $f$ of outage-causing failures be common cause (a bug that affects both replicas; a shared dependency like an exchange gateway or a DNS provider; a config push; a correlated market-data outage). Such outages are not reduced by redundancy, so

$${\text{downtime}}_N\approx (1-A)[f+(1-f)(1-A{)}^{N-1}].$$

As $f\to 1$ the redundancy does nothing. Empirically, common causes dominate: this is why real shops spend as much effort on blast-radius reduction (independent gateways, staggered deploys, circuit breakers) as on replica count. Redundancy is a defence against hardware, not against software.

2.3 Quorum and fencing - guaranteeing at most one primary

The only sound way to prevent split brain is a majority quorum for leadership: with $2F+1$ voters, a leader requires at least $F+1$ votes, so at most one leader can exist while any majority is reachable, and the system tolerates $F$ simultaneous voter failures. Additionally, every primary must hold a fencing token (monotonically increasing lease/epoch) that the resource - the exchange session, the risk gateway - validates, so that a stale primary’s orders are rejected by the resource even if the primary has not noticed it lost leadership. Without fencing, a paused-then-resumed primary can still emit orders; with it, those orders are refused at the wire.

2.4 Failover timing (RTO) and state loss (RPO)

With heartbeat interval $h$ and a “declare the peer dead after $k$ consecutive missed beats” policy, the timeout is $\tau =kh$ and the expected detection time for a failure occurring uniformly in the beat cycle is

$$\mathbb{E}[\text{detect}]=\frac{\tau }{2}+\frac{h}{2}.$$

Total recovery is detection plus promotion (fence the old primary, replay the journal, warm caches, reconnect the sessions, reconcile):

$$\text{RTO}=\mathbb{E}[\text{detect}]+t_{\text{promote}},$$

and the state loss is bounded by the replication lag:

$$\text{RPO}\le \max (\text{replication lag},\text{journal flush interval}).$$

The tension is explicit in $\tau =kh$: a small $\tau$ gives a small RTO but raises the false-failover rate (a GC pause or a transient network stall looks like death), and a false failover means a second primary - precisely the split brain that fencing exists to survive. The standard resolution is a heartbeat timeout tuned to $\sim$3–5 missed beats plus an independent failure detector (a gossip/quorum view) rather than a single peer’s timeout.

2.5 Why replication of strategy state is the hard part

Position, working orders, and P&L accumulate through a sequence of irreversible external events (fills). Replicating the inputs (the journal of market data and order events) is generally cheaper and safer than replicating the outputs: replaying the journal on the standby produces a deterministic state provided the strategy is deterministic - which is exactly why event-driven engines with explicit event logs (see Event-Driven Backtesting Engines) are the right foundation for HA. If the strategy is not deterministic, no amount of replication saves you, and reconciliation becomes the only correctness mechanism.

---

3. Computational Implementation - availability, common cause and failover timing

Stdlib only, deterministic. The script computes single-runtime availability and annual downtime, the independent-replica improvement, the common-cause-corrected figure, and the failover RTO implied by a heartbeat policy.

Read the two stages against each other. The independent model is seductive: two replicas take annual downtime from 3.041 h to 3.8 s. But with a mere 50% common-cause fraction, the realistic figure is 1.521 h/yr - 1,440× worse than the independent model promised, and only $2\times$ better than having no redundancy at all. Meanwhile the failover control plane alone costs RTO = 2.75 s, most of it the expected heartbeat detection delay - far more than the machine failure it is recovering from. HA buys you seconds of availability and pays for it in seconds of latency; that exchange is the whole design problem.

---

4. Failure Modes & First-Principles Breakdowns

1. Split brain. Two primaries trade simultaneously; positions double and neither knows. Mitigation is quorum leadership plus fencing tokens validated at the resource, not mutual politeness between peers.
2. Failover into a stale book. The standby promotes with a book that lags reality and, crucially, believes it. Recovery MUST begin with reconciliation against the venue (drop copy), never with resuming from replicated in-memory state.
3. False failover from an over-tight heartbeat. A GC pause, a log flush, or a transient network stall trips the detector, and now there are two primaries. Tune the timeout, and combine the peer heartbeat with an independent failure detector.
4. Correlated failure ignored. Identical code, identical config, identical dependency, staggered-but-overlapping deploys - the replicas fail together, and the outage is exactly the one HA was bought to prevent (§2.2).
5. Replication of the wrong thing. Replicating positions/P&L rather than the deterministic event journal makes correctness depend on consistency of a derived quantity; if a replication message is lost, the two books disagree silently. Replicate the journal; derive the state.
6. Failover that skips reconciliation. The highest-severity variant of #2: the system comes back fast (great RTO) and wrong (unbounded risk). Fast recovery without a reconciliation gate is worse than a slow, careful one.
7. Un-tested failover. An untested failover path is a hypothesis. The only credible evidence is a game day: kill the primary in production, at a scheduled time, and measure the actual RTO and the reconciliation outcome.
8. Kill switch and failover interacting badly. A kill switch that halts the primary must also halt the standby, or the failover simply resurrects the runaway strategy. Trading-state (RUNNING / HALTED) must be part of the replicated state, and the guard must default to HALTED on promotion until the reconciliation gate clears.

---

5. References

• Kleppmann, Martin, Designing Data-Intensive Applications (O’Reilly, 2017)
• Beyer et al., Site Reliability Engineering (O’Reilly, 2016)
• Ongaro & Ousterhout, In Search of an Understandable Consensus Algorithm (Raft, USENIX ATC 2014)
• Narang, Inside the Black Box, 2nd ed.
• NautilusTrader - Official Documentation (nautilustrader.io)
• BIS / BCBS, Principles for the Sound Management of Operational Risk (2011)

---

6. Connected Graph Bridges

• Back: 05 · Failure Modes · Hub: Index Hub
• Related: Concurrency & Lockless Programming (atomics, memory ordering, the primitives leader election is built from) · Low-Latency Linux & Networking (the OS/network layer whose failures HA absorbs)
• Related: Event-Driven Backtesting Engines (the deterministic event journal that makes replay-based failover sound) · 04 · Risk Guards & Kill Switches (trading-state replication and the HALTED-default on promotion)
• Cross-pillar: Operational Risk (business continuity, loss events from outages) · Quantitative Risk Management
• Base: Probability & Measure Theory (independence, and why it fails) · Numerical Methods