7.3 Tree & Boosting Methods

Basic Prerequisites: Statistics & Inference (bias–variance, model selection) and 05 · Bias–Variance & Validation. (these are the folder-level prerequisites for pages 02–06; page 01 states its own, smaller, entry requirements)

1. Intuition & Practical Objective

Decision trees partition the feature space into rectangles and fit a constant inside each rectangle. That single idea is why trees - and their ensembles, random forests and gradient-boosted trees (XGBoost / LightGBM / CatBoost) - are the default workhorse of tabular quantitative ML. A tree makes no linearity assumption, tolerates outliers (the split only cares about order), natively captures interactions (a deeper path is a conjunction of conditions), and needs no feature scaling. On a factor cross-section where “momentum works only above a volatility threshold” or “value pays only when quality is high”, a linear model must be told the interaction; a tree discovers it.

This folder is a hub: it (a) gives you the fast method/derivation lookup below (job #1), and (b) routes you through six sub-pages that build the intuition from zero, derive the split criterion and the boosting recursion, implement a tree / forest / booster from scratch, name the failure modes that matter in low-SNR finance, and hand you the feature-importance toolkit (MDI/MDA/SFI) and the ensembles/monotonicity extensions.

The one-sentence essence. “Grow many weak, axis-parallel, interaction-aware learners - average them to kill variance (bagging/RF), or add them stagewise along the negative gradient of a loss to kill bias (boosting) - and in the low-SNR financial cross-section this tabular family, not deep nets, is what actually wins.”

2. Mathematical Ground Truth & Lookup

All numbers in the check column were re-executed and reproduced exactly from the runnable code in the sub-pages (numpy only).

Notation: training set $\{(x_i,y_i){\}}_{i=1}^N$, $x_i\equiv (x_{i1},\dots ,x_{ip})$; tree region $R_m$ with leaf count $|T|$ and node sample sizes $N_m$; ensemble size $B$ (RF) or $M$ (boosting); learning rate $\nu$; tree $f(x)={\sum }_m{c}_{m}1(x\in R_m)$.

The critical finance caveat. Trees are low-bias, high-variance and will memorize noise if grown deep; the IC ceiling (a legitimate daily cross-sectional model reaches $R^2\lesssim {\mathrm{IC}}^2\approx 0.25\%$) means depth, learning rate, and the number of trees are not free parameters - they are the whole ball game. Every failure mode in 05 · Failure Modes reduces to this.

3. Computational Implementation - the shared tree engine

Every page in this folder runs on a compact, dependency-free CART engine (numpy only); this is the exact engine the sub-pages call. It reproduces the split on the synthetic panel.

4. Failure Modes & First-Principles Breakdowns

Hub signposts - the full treatment lives in 05 · Failure Modes & Practice. In one line each:

1. Unbounded depth memorizes noise - train MSE keeps falling while test MSE turns up (depth 5 → 8 → 12 gives test MSE $0.3008\to 0.3463\to 0.3680$).
2. MDI is an in-sample, cardinality-biased ranking - a pure-noise continuous feature scored MDI $0.459$ against $0.541$ for the single informative binary feature.
3. MDA under correlated features is a substitution trap - permuting one of two near-duplicates shows almost no drop, so both look irrelevant (AFML Ch 8).
4. Feature selection before the fold split is leakage - screening on the whole sample produced a phantom $+0.0462$ $R^2$ on data whose true $R^2$ is $0.00$ (honest folds gave $-0.1687$).
5. Boosting overfits in low SNR - train error falls monotonically while test error plateaus then rises; $\nu$ and $M$ must be validated out-of-sample.

5. References

• Hastie, Tibshirani & Friedman: The Elements of Statistical Learning (2nd ed., 2009)
• Chen, Tianqi & Guestrin, Carlos: “XGBoost: A Scalable Tree Boosting System,” KDD, 2016
• Ke, Guolin et al.: “LightGBM: A Highly Efficient Gradient Boosting Decision Tree,” NeurIPS, 2017
• López de Prado, Marcos: Advances in Financial Machine Learning (Wiley, 2018)
• Gu, Shihao; Kelly, Bryan; Xiu, Dacheng: “Empirical Asset Pricing via Machine Learning,” RFS 33(5), 2020

6. Connected Graph Bridges

• Foundational base: Statistics & Inference · 05 · Bias–Variance & Validation
• Sibling topics: Financial ML Pitfalls & Low SNR · Purged CV & Backtest Hygiene · Tree-Based Factor Ranking & Purged CV
• Regression base: Feature Engineering & Target Labeling · Fundamental Multi-Factor Models
• Sub-pages (in-folder): 01 From Zero · 02 Decision Trees · 03 Bagging & Random Forests · 04 Gradient Boosting · 05 Failure Modes & Practice · 06 Advanced Extensions

Beginner: start at 01 · Practitioner: start at 05