NLP Text Representation: N-grams, Bag-of-Words & TF-IDF

The core problem: text isn't numbers

Machine learning models operate on vectors of numbers. Text is none of that — it's variable-length sequences of discrete symbols. Before any model (from logistic regression to a transformer) can touch text, something has to turn it into a fixed-size numeric representation. Modern systems do this with learned embeddings, but long before embeddings existed — and still today, as fast, interpretable baselines — practitioners used a family of counting-based techniques: bag-of-words, n-grams, and TF-IDF. Understanding them is still essential, because they clarify exactly what embeddings improve on, and because they remain the right tool for a large share of real-world text problems.

Bag-of-words: throw away order, keep counts

The bag-of-words (BoW) model represents a document as a vector of word counts, completely discarding word order and grammar. Build a vocabulary of every unique word across your corpus, then represent each document as a vector where each entry is how many times that vocabulary word appears in the document.

"The cat sat on the mat" and "the mat sat on the cat" produce the *identical* bag-of-words vector — all that's preserved is which words appeared and how often. This is a real limitation (word order carries meaning), but the simplicity is also the appeal: BoW is fast to compute, easy to interpret, and works surprisingly well for tasks like topic classification and spam detection where word presence matters more than word order.

N-grams: putting a little order back

Pure BoW's biggest weakness is throwing away word order entirely — "not good" and "good" get treated as unrelated to "not" being present, losing the negation. N-grams partially fix this by treating contiguous sequences of $n$ words as single units instead of individual words.

- Unigrams ($n=1$): single words — this is plain bag-of-words
- Bigrams ($n=2$): two-word sequences, e.g. "not good", "very happy"
- Trigrams ($n=3$): three-word sequences, e.g. "not very good"

Including bigrams lets a model directly learn that "not good" behaves very differently from "good" — information unigrams alone can't capture. The cost: vocabulary size explodes combinatorially as $n$ grows, and most higher-order n-grams appear only once or twice in a corpus, adding sparse, noisy dimensions. In practice, unigrams plus bigrams (sometimes trigrams) is the usual sweet spot.

The problem with raw counts: common words dominate

Raw word counts have an obvious flaw: words like "the", "is", and "of" appear constantly in every document and contribute nothing to distinguishing one document from another, yet they dominate the count vector numerically. You could manually remove these stop words, but a more principled fix is to reweight every term by how informative it actually is — which is exactly what TF-IDF does.

TF-IDF: weighting words by how distinctive they are

TF-IDF (Term Frequency–Inverse Document Frequency) rescales raw word counts by how rare that word is across the whole corpus. The intuition: a word that appears often in one document but rarely elsewhere is highly distinctive of that document's content; a word that appears constantly everywhere (like "the") is not informative, no matter how often it shows up.

$\text{TF}(t, d) = \frac{\text{count of term } t \text{ in document } d}{\text{total terms in } d}$

$\text{IDF}(t) = \log\left(\frac{N}{\text{number of documents containing } t}\right)$, where $N$ is the total number of documents

$\text{TF-IDF}(t, d) = \text{TF}(t, d) \times \text{IDF}(t)$

A word that's frequent in a document (high TF) but rare across the corpus (high IDF) gets a high score. A word that's everywhere (low IDF, since nearly every document contains it) gets pushed toward zero regardless of its local frequency.

Bag-of-words vs. TF-IDF vs. embeddings

What's next

These counting-based methods are the baseline that dense embeddings were built to surpass — embeddings capture semantic similarity that no amount of word counting can. For the architecture that generates today's best embeddings and processes text sequences directly, see transformers.