TF-IDF Deep Dive
Why raw word counts fail
If you simply count word occurrences per document, the words that dominate every document — 'the', 'is', 'and' — swamp the genuinely distinguishing words. TF-IDF (Term Frequency–Inverse Document Frequency) fixes this by weighting each word's frequency within a document against how rare it is across the whole corpus, so common words are downweighted and distinctive, document-specific words are upweighted.
The formula, term by term
Term frequency $\text{tf}(t, d)$ measures how often term $t$ appears in document $d$, often normalized by document length: $\text{tf}(t,d) = \frac{\text{count}(t, d)}{\sum_{t' \in d} \text{count}(t', d)}$. Inverse document frequency measures how rare the term is across the full corpus of $N$ documents: $\text{idf}(t) = \log\left(\frac{N}{1 + \text{df}(t)}\right)$, where $\text{df}(t)$ is the number of documents containing $t$. The $+1$ in the denominator (smoothing) prevents division by zero for terms absent from the corpus. The final score is the product: $\text{tfidf}(t, d) = \text{tf}(t,d) \times \text{idf}(t)$. A word that appears frequently in one document (high tf) but rarely elsewhere (high idf) gets the highest score — exactly the words that characterize that document.
import math
from collections import Counter
def compute_tfidf(corpus: list[str]) -> list[dict]:
tokenized = [doc.lower().split() for doc in corpus]
N = len(tokenized)
# document frequency: how many docs contain each term at least once
df = Counter()
for doc in tokenized:
for term in set(doc):
df[term] += 1
idf = {term: math.log(N / (1 + freq)) for term, freq in df.items()}
tfidf_docs = []
for doc in tokenized:
counts = Counter(doc)
total = len(doc)
tfidf_docs.append({
term: (count / total) * idf[term]
for term, count in counts.items()
})
return tfidf_docs
corpus = [
"the cat sat on the mat",
"the dog chased the cat",
"machine learning models need data",
]
scores = compute_tfidf(corpus)
# "machine", "learning", "data" score highly in doc 3 (high tf, high idf);
# "the" scores near zero everywhere (appears in nearly every doc).Variants that matter in practice
Real implementations (e.g., scikit-learn's TfidfVectorizer) add refinements: sublinear tf scaling ($1 + \log(\text{tf})$) dampens the effect of very high counts, since the 20th occurrence of a word matters much less than the 2nd. L2 normalization of the final vector makes TF-IDF vectors directly comparable via cosine similarity regardless of document length. Smooth IDF adds 1 inside the log to avoid negative weights for terms appearing in every document. These details seem minor but materially change downstream similarity rankings and classifier weights — always check which smoothing variant a library uses before comparing scores across tools.
| Scenario | TF-IDF behavior | Implication |
|---|---|---|
| Word in every document | idf → 0, term effectively ignored | Automatic stopword-like suppression |
| Word unique to one document | Very high idf | Strong signal, but risk of overfitting to rare terms |
| Very long documents | Raw tf inflated without normalization | Must normalize tf by document length |
| Short queries vs. long documents | tf-idf magnitude mismatch | Cosine similarity needed, not raw dot product |
TF-IDF is a retrieval algorithm, not just a feature engineering trick
TF-IDF has no notion of meaning
What's Next?
TF-IDF's weakness — no sense of word meaning — is exactly the gap that techniques built around co-occurrence statistics and latent structure were designed to close.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.