8.9.2 The Research-to-Production Lifecycle & Deployment

Basic Prerequisites: 01 · From Zero and Event-Driven Backtesting Engines.

---

1. Intuition & Practical Objective

A strategy does not “go live.” It graduates, one gate at a time, and each gate exists to make a specific class of mistake expensive to pass rather than free. Deployment is the discipline that decides how much capital a change is allowed to risk before it has earned more.

The core asymmetry: a code change carries no information about whether it is correct. The developer believes it is; the developer is the least reliable witness. So the deployment system must convert “believe” into “have observed.” That is what a canary is: you give the new build a small amount of real money and watch, and only increase exposure when the observed behaviour matches the predicted behaviour.

Two deployment patterns, in increasing order of safety and cost:

• Blue-green: two complete environments (blue = current, green = new). You bring green up cold, verify it, then switch 100% of traffic in one step. Rollback is a switch back. Simple, low-latency-to-deploy, but the first instant of exposure is 100% - you find out the build was bad at full size.
• Canary: route a small weight $w$ (1%, 5%, 25%…) to the new build and ramp. You find out the build is bad at 1% of the cost. This is the pattern trading systems should use, because the downside of a bad build is measured in money, not in user-visible errors.

The one-sentence essence. “Deploy is a risk decision, not a technical event: the question is never ‘does the new code work?’ but ‘how much money is exposed before we would notice it does not?’ - so you ramp exposure (canary), define the metric that triggers rollback before deploying, and automate the rollback so it does not depend on a human being awake.”

---

2. Mathematical Ground Truth & Derivations

2.1 Exposure-hours: the quantity a rollout minimises

Model a bad build as producing a loss at rate $L$ (dollars per hour) whenever it is live at full traffic. If a rollout assigns traffic weight $w(t)\in [0,1]$ over a horizon $[0,T]$, the loss incurred is

$$\mathcal{L}=L{\int }_0^{T}w(t)dt=L\cdot \mathcal{E},\mathcal{E}={\int }_0^{T}w(t)dt(\text{exposure-hours}).$$

For a big-bang deploy, $w(t)=1$ until detection at $t=D$, so $\mathcal{E}=D$. For a canary with $K$ stages of weights $w_k$ held for $h_k$ hours,

$${\mathcal{E}}_{\text{canary}}=\sum _{k=1}^K{w}_k{h}_k,$$

and the rollout aborts at the first stage where the cumulative loss exceeds the rollback threshold $\Theta$:

$$k^{\ast }=\min \{k:L\sum _{j\le k}{w}_j{h}_j\ge \Theta \}.$$

The reduction factor ${\mathcal{E}}_{\text{big}}/{\mathcal{E}}_{\text{canary}}$ is the entire value proposition of the pattern, and it is monotone in how low the first weight is.

2.2 The rollback-trigger design problem

The threshold $\Theta$ is a trade-off, not a constant. Roll back too eagerly and good deploys die to noise (high false-abort rate, release velocity collapses); too late and a bad deploy runs long. Under normal P&L noise with standard deviation ${\sigma }_{\Delta }$ per window and a rollback rule “abort if cumulative deviation $>k{\sigma }_{\Delta }$”, the false-abort probability per window is $2(1-\Phi (k))$ - the same control-chart arithmetic as 03 · Monitoring & Alerting.

2.3 Expected cost of a deploy programme

With per-deploy bad-build probability $p$, the expected loss per deploy is $pL\mathcal{E}$, and over $N$ deploys per year the annual expected loss is $pNL\mathcal{E}$. Note how the terms separate: you control $N$ (release cadence), $\mathcal{E}$ (rollout pattern), and - only through testing - $p$. Since the framework, engine, and strategy are all deployed through the same pipeline, $\mathcal{E}$ is the highest-leverage knob you own, and it is pure process.

2.4 Blue-green vs canary

Blue-green is the $K=1$ special case: weights $w_1=1$, aborted at $h_1=D$. Canary dominates it on $\mathcal{E}$ whenever the first-stage weight is small, at the cost of a longer time-to-full-traffic and slightly higher state-complexity (two versions live simultaneously, so positions/P&L must be attributable per version - a bookkeeping burden the rollout creates).

---

3. Computational Implementation - the canary ramp with automatic rollback

Stdlib only, deterministic. The simulator ramps traffic $1\%\to 5\%\to 25\%\to 100\%$ against a bad build, accumulates the loss at each stage, and trips an automatic rollback the moment the cumulative loss exceeds the threshold. It then compares the realised loss against the same bad build deployed big-bang.

The headline: the same bug, the same detection speed, a $12.9\times$ difference in realised loss - because the canary let the loss announce itself at 1% of exposure instead of 100%. Note also that the canary’s Time under a bad build (1.50 h) is shorter than big-bang’s (2.00 h) despite more stages, because the abort is automated and the ramp front-loads the cheap observation.

---

4. Failure Modes & First-Principles Breakdowns

1. The canary that is not canaried. Routing “10% of signals” while the strategy shares one position and one risk budget means the risk of the new build is already 100%. True canarying requires separate capital, separate risk limits, and separate P&L attribution per version - otherwise you have deployed the new build at full size and merely labelled it.
2. Rollback that needs a human. An automatic rollback bound to an automated metric (cumulative loss > Θ) fires in seconds; a rollback that pages a human fires in tens of minutes. If the trigger path includes a person, model it as $\Theta \to \infty$ for the purpose of $\mathcal{E}$.
3. Deploy-time state drift. Restarting the system resets in-memory state; if the strategy’s state (position, working orders, VWAP accumulators) is not recovered from the broker, the new process starts believing it is flat. This is the deployment-flavoured version of 05 · Failure Modes’ stale-position bug.
4. Configuration as an untracked dependency. “The deploy was fine, somebody changed a limit.” Version the limits with the code - a limit edited out-of-band is an unreviewed code change to the one component whose job is to say no.
5. Promoting on P&L alone. A canary that matches P&L but shows double the intended order rate is broken in a way that has not cost money yet. Gate promotion on the full metric set (rate, reject ratio, latency, reconciliation status), not just the bottom line.

---

5. References

• Davey, Building Winning Algorithmic Trading Systems
• Narang, Inside the Black Box, 2nd ed.
• Beyer et al., Site Reliability Engineering (O’Reilly, 2016)
• Humble & Farley, Continuous Delivery (Addison-Wesley, 2010)
• NautilusTrader - Official Documentation (nautilustrader.io)

---

6. Connected Graph Bridges

• Back: 01 · From Zero · Hub: Index Hub
• Next: 03 · Monitoring & Alerting (the metrics that trigger rollback) · 05 · Failure Modes
• Related: 04 · Risk Guards & Kill Switches (the last-line limit if the rollout trigger fails) · FIX · Session Management (connectivity during a rolling restart)
• Base: Backtesting Hygiene