7.9.1 From Zero

Basic Prerequisites: Statistics & Inference (correlation, regression, the t-statistic). No prior portfolio theory or ML implementation needed.

---

1. Intuition & Practical Objective

Here is the single most important idea in this whole folder: machine learning and portfolio construction are two different jobs, and the strategy is decided at the seam between them.

• ML’s job: turn features (momentum, value, alt-data, analyst revisions…) into a forecast - a number, per asset, per period, that says “this asset will go up/down by about this much.”
• Portfolio construction’s job: turn those forecasts into positions - how much capital to put on each asset, subject to your risk budget and constraints.
• The seam: the position is the product of (forecast quality) × (allocation quality). A perfect forecast and a naive allocation can still lose money. A perfect allocation fed garbage forecasts loses money too. You cannot fix one stage by being perfect at the other.

The practical objective of the ML-for-portfolio discipline is to stop treating these as two disconnected black boxes and instead design the interface: weight forecasts by how reliable they are, cut their variance by ensembling, and feed the optimizer a covariance matrix you actually trust.

---

2. Mathematical Ground Truth & Derivations

2.1 The pipeline as a composition of maps

Let $x_t$ be features and $y_{t+1}$ the forward return. The full chain is:

$$x_t\stackrel{\text{model }g}{\to }{\hat{y}}_{t+1|t}\stackrel{\text{position rule }p}{\to }{w}_t\stackrel{\text{market}}{\to }{r}_{t+1}=w_t^{\top }{y}_{t+1}.$$

Three errors enter here, and they multiply rather than add:

1. Model error $e_t^g={\hat{y}}_{t+1|t}-y_{t+1}$ - how wrong the forecast is.
2. Position-rule error $e_t^p=w_t-p({\hat{y}}_{t+1|t})$ - how badly the position rule uses the forecast it was given.
3. Covariance/allocation error - how far the estimated risk model is from the true one.

The realized PnL error is, to first order, the sum of the products of these, because the position and the realized return both carry estimation error:

$$r_{t+1}-w_t^{\top }\mathbb{E}[y_{t+1}]=\underset{\text{forecast error}\times \text{leverage}}{\underbrace{p(\hat{y}{)}^{\top }{e}^g}}+\underset{\text{position error}}{\underbrace{(w-p(\hat{y}){)}^{\top }y}}+\dots$$

This is why the discipline exists: each stage’s error is amplified by the next stage’s gain (leverage, notional). A 1% forecast error on a 5× levered book is a 5% PnL error, before the optimizer adds its own.

2.2 Bias, variance, noise - where model error comes from

For a model $\hat{f}$ trained to approximate $y=f(x)+\epsilon$ (López de Prado AFML Ch 6; ESL §7.3):

$$\mathbb{E}[(y-\hat{f}(x){)}^2]=(\mathbb{E}[\hat{f}]-f{)}^2+\mathrm{Var}(\hat{f})+{\sigma }_{\epsilon }^2.$$

• Bias - the model is too simple to see real structure (underfit).
• Variance - the model is so sensitive that small training changes produce wildly different forecasts (overfit).
• Noise - the irreducible ${\sigma }_{\epsilon }^2$; no model can explain it.

Finance is a low-signal setting: ${\sigma }_{\epsilon }^2$ dominates, so variance is your enemy and ensembling (03 · Ensembling) is the natural medicine. The bias–variance knob is the dial the whole pillar turns.

2.3 Forecast quality is not portfolio quality

Two different summary statistics, frequently conflated:

• Information Coefficient (IC) - the correlation between forecast and realized return, $\mathrm{IC}=\mathrm{corr}(\hat{y},y)$. It measures forecast quality.
• Position IC - the correlation between the position you actually take and realized return, $\mathrm{corr}(w,y)$. It measures portfolio quality.

The first can be excellent and the second terrible (e.g., you forecast correctly but size the bet wrong - see 02 · Forecasts→Positions). Portfolio construction is the machine that converts IC into position IC, and it does so with friction.

---

3. Computational Implementation - a minimal end-to-end seam

Stdlib + numpy. This composes a model, a position rule, and a covariance into one PnL number, so you can see the seam.

Read the rows carefully: the forecast IC is identical (0.0674) under both position rules - correlation is scale-invariant, so it cannot tell you anything about your position. The seam is the lever that does: the position rule governs the scale of exposure (half the notional → half the PnL and half the risk, same Sharpe). The strategy’s quality (IC, Sharpe) is set by the forecast; the seam decides how much of that edge you are willing to risk. That is the essence of portfolio construction and of this folder.

---

4. Failure Modes & First-Principles Breakdowns

1. Optimizing the stages in isolation. Tuning the model to maximize in-sample IC, and separately the optimizer to minimize in-sample variance, double-dips the same test set - you are effectively selecting on the seam, not each stage. The two-stage objective is ${\max }_{g,p}\mathrm{corr}(p(g(x)),y)$, not ${\max }_g\mathrm{IC}(g)+{\max }_p\dots$.
2. Confusing forecast error with position error. A forecast with great IC but wrong scale, sign-of-uncertainty, or timing produces a bad position. Measure portfolio metrics (position IC, net PnL, turnover), not just model metrics.
3. Ignoring that the market adapts. Your forecast edge decays precisely because others trade it (see Financial-ML Pitfalls & Low SNR). The seam must be re-estimated on a rolling basis, not fixed once.
4. Thinking more signal always helps. Below a reliability threshold, adding a noisy forecast to the mix can reduce combined IC (inverse-variance weighting fixes this - see 02).

---

5. References

• López de Prado, Advances in Financial Machine Learning (2018)
• Hastie, Tibshirani & Friedman, The Elements of Statistical Learning (2009)
• DeMiguel, Garlappi & Uppal, “Optimal Versus Naive Diversification,” RFS (2009)

---

6. Connected Graph Bridges

• Forward: 02 · Forecasts→Positions · 03 · Ensembling Models
• Theory: Statistics & Inference · Linear Algebra & Matrices
• Sibling signal source: Factor Investing & Factor Timing · Portfolio Optimization