7.4.2 Bag-of-Words & the Loughran–McDonald Finance Dictionary

Basic Prerequisites: 01 · From Zero (why counts are a signal) and Statistics & Inference.

---

1. Intuition & Practical Objective

The bag-of-words model is the workhorse of financial text analysis: turn each document into a count vector, then feed those vectors to any regression or classifier. Its two engineering problems are (a) ubiquity - “the”, “company”, “fiscal” appear in every document and carry no discriminating power - and (b) dictionary calibration - a word’s emotional valence is not universal.

This page delivers both fixes, and they are the foundation of every downstream result:

1. TF–IDF rescaling, which down-weights ubiquitous terms and up-weights terms that are rare in the corpus but frequent in a specific document - the standard “which words actually distinguish this text” feature (ESL’s spam classifier uses exactly this family of term features; Ch 18 frames the $p\gg N$ side).
2. The Loughran–McDonald (LM) dictionary, the finance-specific sentiment lexicon (negative/positive/uncertainty/litigious/constraining/strong-modal word lists) hand-built from 10-Ks precisely because general dictionaries misclassify accounting words.

The objective is one sentence: you need term weights that survive contact with an SEC filing, and a sentiment list that speaks accounting, not newsrooms.

---

2. Mathematical Ground Truth & Derivations

TF–IDF. For document $d$ and term $t$ with count $x_{d,t}$:

$$\text{tf-idf}(t,d)=\underset{\text{term frequency (normalized)}}{\underbrace{\frac{x_{d,t}}{{\sum }_{t^{\prime }}{x}_{d,t^{\prime }}}}}\times \underset{\text{inverse document frequency (smoothed)}}{\underbrace{(\log \frac{N+1}{d{f}_t+1}+1)}},$$

with $N$ documents and $d{f}_t$ = number of documents containing $t$. The smoothing $+1$ keeps the IDF positive for a term in every document (so rare-vs-common ranking stays stable) and matches the classic implementation (Jurafsky & Martin Ch 6; sklearn’s TfidfTransformer). Intuition: a term scores high only if it is frequent in this document and unusual across the corpus - “impairment” in one 10-K among a hundred that never mention it is exactly the kind of term that should spike.

The document-distance view. With term vectors ${\mathbf{v}}_d=(\text{tf-idf}(t,d){)}_t$, two documents are compared by cosine similarity

$$\text{cos}({\mathbf{v}}_d,{\mathbf{v}}_{d^{\prime }})=\frac{{\mathbf{v}}_d\cdot {\mathbf{v}}_{d^{\prime }}}{\Vert {\mathbf{v}}_d\Vert \Vert {\mathbf{v}}_{d^{\prime }}\Vert },$$

which is the primitive behind both the Cohen–Malloy–Nguyen “did the firm change its language?” metric ($1-\cos$) and nearest-neighbor retrieval on filings.

Why general dictionaries fail (Loughran–McDonald 2011). Loughran–McDonald measured the misfit directly: the Harvard IV-4 and Henry negative lists classify large fractions of negative-tagged words that are actually accounting terms with neutral valence. Among the words Harvard labels negative that recur in 10-Ks are “liability”, “tax”, “cost”, “capital”, “gross”, “share”, “unrealized” - none of which is management expressing pessimism. A model built on such a list counts the accounting itself as sentiment. The LM lists re-sort 10-K word-frequency counts into a finance-native lexicon. The measurable consequences: sentiment scores built with generic dictionaries are biased (mechanically more “negative” the more accounting-specific the document), and the predictive content - the whole point - is diluted or inverted.

---

3. Computational Implementation - TF–IDF matrix and LM negative proportion from scratch

Builds the TF–IDF feature matrix over a four-document 10-K-like corpus and reports the LM negative-word proportion per document (a tiny stand-in for the full LM list, enough to show the mechanics).

Two lessons in the numbers. First, TF–IDF isolates the discriminating terms: “impairment”/“loss” spike in doc0, “risk”/“restructuring” in doc3, “demand”/“revenue” in doc1 - each term concentrates on the document that actually discusses it. Second, the LM proportion is meaningful because it is a proportion: doc0 and doc3 are both “negative” (0.333) even though their vocabulary is entirely different - the lexicon compresses different phrasing into a comparable scalar.

---

4. Failure Modes & First-Principles Breakdowns

1. Raw counts dominate by length. Without IDF (or length normalization) the most frequent words win and a long boilerplate filing drowns the signal. First principle: information is per-word, so normalize.
2. Dictionary valence mismatch. “Liability”, “tax”, “cost”, “gross”, “unrealized” are LM-vs-Harvard misfits - counting them as negative fabricates pessimism and biases the feature (Loughran–McDonald 2011). First principle: a word’s valence is conditional on the domain, so the lexicon must be built on the domain corpus.
3. Sparse high-dimensional features invite overfitting. $|V|$ is thousands; a few hundred filings give $p\gg N$. The TF–IDF matrix is the exact input ESL Ch 18 warns about: without regularization or feature screening, in-sample fit is trivial and out-of-sample meaningless (measured in 05).

---

5. References

• Loughran, Tim & McDonald, Bill, “When Is a Liability Not a Liability? Textual Analysis, Dictionaries, and 10-Ks,” Journal of Finance 66(1), 2011
• Jurafsky, Dan & Martin, James H., Speech and Language Processing (3rd ed.)
• Hastie, Tibshirani & Friedman, The Elements of Statistical Learning
• Gentzkow, Kelly & Taddy, “Text as Data,” JEL 57(3), 2019

---

6. Connected Graph Bridges

• Back: 01 · From Zero · Index Hub
• Forward: 03 · Sentiment & Tone · 04 · Embeddings & Transformers
• Data provenance: Data Sources & Corporate Data (the filings these documents come from)