7.9 ML for Portfolio Construction

Basic Prerequisites: Linear Algebra & Matrices (inverses, condition numbers) and Statistics & Inference (covariance, correlation, OLS). Prior exposure to Mean–Variance Optimization is strongly recommended; ML itself is treated as a black box you have already trained. (these are the folder-level prerequisites for pages 02–06; page 01 states its own, smaller, entry requirements)

1. Intuition & Practical Objective

Machine learning does not build portfolios - it produces forecasts. Portfolio construction is the second stage: it takes those forecasts and turns them into positions. The strategy lives and dies at that interface. Every unit of forecast error, and every unit of estimation error in the covariance matrix, compounds through the position-sizing and allocation steps. A brilliant model bolted onto a naive optimizer, or a great optimizer fed garbage forecasts, both lose the same way.

This folder is the Pillar 7 ML-for-portfolio topic-folder and it is a hub: (a) it gives the fast formula lookup below (job #1), and (b) it routes you through six sub-pages that walk you from raw intuition through combining forecasts into positions, ensembling model outputs, ML for covariance/factor estimation, the failure modes of stacking ML on an optimizer, and the advanced extensions (meta-labeling, HRP/HERC, denoised covariance, RL allocation).

The one-sentence essence. “ML converts features into a forecast; portfolio construction converts that forecast into a position. The edge is decided at the seam between the two - so weight your model forecasts by reliability, ensemble them to cut variance, shrink or denoise the covariance you feed the optimizer, and never let a small estimation error become a large position error.”

The audience arc: beginner starts at 01 (the pipeline) → 02 (forecasts→positions) → 03 (ensembling). Practitioner/graduate continues to 04 (covariance & factors) → 05 (failure modes) → 06 (advanced extensions). Throughout, the folder cross-links hard to Pillar 5 (Portfolio Optimization) and Pillar 1 (Factor Investing & Factor Timing).

2. Mathematical Ground Truth & Derivations

Quick-Reference Lookup (job #1). Formulas transcribed from López de Prado, Advances in Financial Machine Learning (2018, Ch 6 & 16), Machine Learning for Asset Managers (2020, Ch 2, 4 & 7), and Granger & Ramanathan (1984); the bias–variance decomposition and bagging variance formula are cross-checked against Hastie, Tibshirani & Friedman, ESL (2009) §7.3 and eq. 15.1. Every number in the check column was re-executed and reproduced exactly from the verified corpus (see §3).

Notation: $f_i$ forecast of model $i$, ${\sigma }_i^2$ variance of model $i$‘s forecast error, $\mu$ vector of expected returns, $\Sigma$ covariance matrix, $r$ realized return, ${\rho }_{ij}$ correlation, $B$ number of ensemble members, $\Phi (\cdot )$ standard-normal CDF.

The one number that predicts the whole folder. Markowitz’s curse: to estimate a non-singular $N\times N$ covariance you need $\frac{1}{2}N(N+1)$ IID observations (López de Prado 2018, Ch 16.3). For $N=30$ that is 465 independent daily returns ≈ 2 years of non-overlapping data. Below that, the sample covariance is ill-conditioned and its inverse amplifies estimation noise by $O(\Delta {\lambda }_i/{\lambda }_i^2)$ - largest exactly where the eigenvalues are smallest. That is the structural reason the naive $1/N$ portfolio so often beats mean-variance out-of-sample (DeMiguel et al., 2009).

3. Computational Implementation - the pipeline engine

This runs on numpy only and reproduces the verified numbers above: combine three forecasts into a position, size a bet from a probability, shrink a covariance, and compute an HRP allocation. (Output annotated inline.)

4. Failure Modes & First-Principles Breakdowns

Hub signposts - the folder’s full failure-mode analysis lives in 05 · Failure Modes & Practice. In one line each:

1. Error compounding across the seam - forecast error and covariance-estimation error are multiplied through the optimizer, not added; a tiny per-input error can become a large position error.
2. Overfitting the ensemble - fitting combination weights (Granger–Ramanathan OLS, stacking) on the same data you evaluate on inflates in-sample IC and collapses out-of-sample (the $+0.2144$ vs $+0.2321$ gap above is exactly this).
3. Forecast-to-position mismatch - a good forecast and a good position are different objects; ignoring reliability (bet sizing, uncertainty) turns a correct signal into a losing bet.
4. Markowitz’s curse - inverting an ill-conditioned sample covariance makes the optimizer bet largest on the assets it knows least (eff-N collapses from 30 to 3.5).
5. Stacking ML on a fragile optimizer - ML inputs are more accurate but still errorful; if the optimizer amplifies input errors, ML inputs into mean-variance inherit the same instability.

5. References

• López de Prado, Marcos: Advances in Financial Machine Learning (2018)
• López de Prado, Marcos: Machine Learning for Asset Managers (2020)
• Granger, C. W. J. & Ramanathan, R.: “Improved Methods of Combining Forecasts,” J. Forecasting 3(2):197–204, 1984
• López de Prado, Marcos: “Building Diversified Portfolios That Outperform Out of Sample,” J. Portfolio Management 42(4):59–69, 2016
• López de Prado, Marcos: “A Robust Estimator of the Efficient Frontier,” SSRN 3469961, 2019
• Hastie, Tibshirani & Friedman: The Elements of Statistical Learning (2nd ed., 2009)
• DeMiguel, Garlappi & Uppal: “Optimal Versus Naive Diversification,” RFS 22(5):1915–1953, 2009
• Ledoit & Wolf: “Improved Estimation of the Covariance Matrix of Stock Returns,” J. Empirical Finance 10(5):603–621, 2003

6. Connected Graph Bridges

• Pillar 5 (portfolio construction): Modern Portfolio Theory & Mean–Variance · Hierarchical Risk Parity (HRP) · Covariance Shrinkage & RMT Denoising · Robust Optimization · Black–Litterman · Kelly Criterion & Bet Sizing
• Pillar 1 (signals): Factor Investing & Factor Timing (where the forecasts come from)
• Pillar 7 (ML stack): Tree & Boosting Methods (the ensembling machinery) · Purged CV & Backtest Hygiene (how you measure the seam) · Financial-ML Pitfalls & Low SNR · Regime Classification
• Sub-pages (in-folder): 01 From Zero · 02 Forecasts→Positions · 03 Ensembling Models · 04 ML for Covariance & Factors · 05 Failure Modes · 06 Advanced Extensions

Beginner: start at 01 · Practitioner: start at 05