8.9 Production Trading Systems

Basic Prerequisites: Event-Driven Backtesting Engines (you must be able to replay time honestly before you can run time live) and FIX & Exchange Connectivity (the wire the orders actually travel on). Working Python. (these are the folder-level prerequisites for pages 02–06; page 01 states its own, smaller, entry requirements)

1. Intuition & Practical Objective

A backtest answers one question: is there an edge? A production trading system answers a different and harder one: does the edge survive contact with reality, and when it stops working - or the code misbehaves - what stops the loss?

Everything that makes a strategy “production” is a negative engineering discipline. Research code optimises for a number; production code optimises for the absence of a catastrophic number. That inversion is the whole subject. The strategy is perhaps 10% of a production system; the other 90% is the machinery that limits how much damage a wrong strategy can do: pre-trade risk guards, position and cash reconciliation, monitoring and alerting, deployment control, incident response, and failover.

This folder is the topic-hub for production trading systems in Kwant-Atlas. It (a) gives you the fast lookup below - the hub’s job #1, a consolidated table of the limits every live system must enforce - and (b) routes you to six sub-pages that walk from raw intuition through the lifecycle, monitoring, risk guards, failure modes, and high availability.

The one-sentence essence. “A production trading system is a risk-controlled state machine that happens to also trade: it must know its own position at all times, refuse any action that would breach a limit, detect the violations it did not anticipate, and be able to go flat and disconnect within a bounded time - no matter what the strategy, the market, or the network does.”

The three laws of production (each a first principle, not a preference):

1. The market does not pause while you debug. Unlike a web service, you cannot take the trading system down for maintenance and keep your risk. The system must have a bounded, pre-tested path to flat and disconnected - a kill switch - that works when the strategy is broken.
2. Risk limits are the only code whose job is to say no. They must live outside the strategy, be always-on, be unreachable by strategy configuration, and be tested as adversarially as the order path itself. A limit you can disable from the strategy is not a limit.
3. You cannot trust a number you cannot reconcile. Your internal position, cash, and P&L are beliefs; the broker/clearing feed is the truth. Until the two are reconciled, you do not know your risk - you know your hope.

2. Mathematical Ground Truth & Lookups

Quick-reference lookup (job #1). These are the canonical limit equations, with the verified numeric check produced by the runnable dashboard in §3 and by the sub-pages (each figure reproduced exactly).

Notation: $q_i$ signed position in instrument $i$, $P_i$ price, $P_{\text{nbbo}}$ the quote mid, $q$ order quantity, $P$ order price, $E_t$ equity, $E_{\text{peak}}$ running equity high-water mark, $b$ token-bucket balance, $R$ refill rate, $B$ bucket capacity, $\tau$ heartbeat timeout.

Critical caveat. These thresholds are conventions, not physics. The correct $G_{\max }$ depends on your capital, your liquidation horizon, and the venue’s margin rules; the correct heartbeat $\tau$ trades detection latency against false failovers. The durable results are the structural ones: a limit must exist per layer (order, book, P&L, rate, wire), and redundancy without common-cause analysis is a story you tell yourself.

The lifecycle lookup. Every stage of the research-to-production arc has one gate and one artifact. If you cannot name both, you are not ready for the stage.

3. Computational Implementation - the consolidated limit dashboard

This is the hub’s capstone: one deterministic, standard-library-only check that evaluates every book-level limit in the table above at once. It is exactly the shape of the pre-trade/at-trade check that must run between the strategy and the wire.

Read it as the system’s reflex: gross exposure is $104.5\%$ of cap, so the correct response is block new orders and flatten toward the limit - not “warn the trader.” A limit that only warns is a limit that will be crossed again tomorrow.

4. Failure Modes & First-Principles Breakdowns

Hub signposts - the folder’s fault analysis lives in 05 · Failure Modes & Practice. In one line each:

1. Runaway orders - a logic bug, a retry loop, or a lost session state emits orders far faster than intended; without a rate limit and a kill switch the loss compounds at wire speed (this is the Knight Capital mechanism, and the reason production-risk-guards-and-kill-switches exists as a flat page).
2. Stale positions - your internal book diverges from the broker’s because a fill was dropped, a cancel was mis-acknowledged, or a session died mid-order; you then compute risk on a fiction. Only reconciliation catches it.
3. Silent failures - the monitor that stopped monitoring, the alert that fired into an unread channel, the reconciliation that has been failing for three days. Dependence on a control without a control on the control is the deepest failure mode in the folder.

5. References

• Narang, Rishi K.: Inside the Black Box: A Simple Guide to Quantitative and High-Frequency Trading (2nd ed., Wiley; 3rd ed. 2024 retitled A Simple Guide to Systematic Investing)
• Davey, Kevin J.: Building Winning Algorithmic Trading Systems (Wiley, 2014)
• Cartea, Jaimungal & Penalva: Algorithmic and High-Frequency Trading (Cambridge, 2015)
• Carver, Robert: Systematic Trading (Harriman House, 2015)
• NautilusTrader - Official Documentation (nautilustrader.io)
• SEC Rule 15c3-5 - Risk Management Controls for Brokers or Dealers with Market Access
• Almgren & Chriss (2000), Optimal Execution of Portfolio Transactions, Journal of Risk
• Beyer et al.: Site Reliability Engineering (O’Reilly, 2016)

6. Connected Graph Bridges

• Sibling topic: Event-Driven Backtesting Engines (the same event loop, replayed on live data - the engine a production system grows out of)
• Sibling topic: FIX & Exchange Connectivity (the order-state machine and session recovery that reconciliation depends on)
• Sibling topic: High-Performance C++ for Trading (the pre-trade risk check lives inside the tick-to-trade latency budget)
• Sibling topic: Concurrency & Lockless Programming (the kill-switch flag must be visible to every thread without a lock)
• Cross-pillar, risk: Quantitative Risk Management · Operational Risk (production incidents are operational risk events) · VaR & Expected Shortfall (what the limits are a crude, fast proxy for)
• Cross-pillar, execution: Market Microstructure & Order Types (collars, order types, cancel-on-disconnect)
• Base: Numerical Methods (floating-point comparison, tolerance design in reconciliation) · Low-Latency Systems Architecture
• Single-page overview (superseded thematically by this folder): Production Risk Guards & Kill Switches
• Sub-pages (in-folder): 01 From Zero · 02 Lifecycle & Deployment · 03 Monitoring & Alerting · 04 Risk Guards & Kill Switches · 05 Failure Modes · 06 Advanced Extensions

Beginner: start at 01 · Practitioner: start at 05

• Absolute beginner (no trading-systems background): 01 · From Zero - why a profitable backtest is not a tradable system.
• Lifecycle + operations (undergrad / job-seeking): 02 · Lifecycle & Deployment → 03 · Monitoring & Alerting → 04 · Risk Guards & Kill Switches.
• Robustness (practitioner / graduate): 05 · Failure Modes → 06 · Advanced Extensions.
• Forward links: Operational Risk · Concurrency & Lockless Programming · Stress Testing & Scenario Analysis