Topic Modeling: LDA Explained

9 min read

Learn

Quick Reading

Estimated 9 mins

Prereq

Intermediate

Basic ML concepts helpful

Interactive

Static Playbook

Static guide & reference tables

Finding themes without labels

Given thousands of unlabeled documents, topic modeling discovers the latent thematic structure spanning the collection — groups of words that tend to co-occur, interpreted as 'topics.' Latent Dirichlet Allocation (LDA), introduced by Blei, Ng, and Jordan in 2003, is the foundational probabilistic topic model: it assumes every document is a mixture of a small number of topics, and every topic is a probability distribution over the vocabulary.

The generative story

LDA is defined by how it imagines documents were generated. For each document $d$: first draw a topic distribution $\theta_d \sim \text{Dirichlet}(\alpha)$ — a vector over $K$ topics summing to 1 (e.g., 70% 'sports', 30% 'finance'). Then for each word position in the document, draw a topic assignment $z \sim \text{Multinomial}(\theta_d)$, and draw the actual word $w \sim \text{Multinomial}(\phi_z)$, where $\phi_z$ is topic $z$'s own distribution over the entire vocabulary (also drawn once per topic from $\text{Dirichlet}(\beta)$). The Dirichlet priors $\alpha$ and $\beta$ control sparsity: small $\alpha$ favors documents concentrated in few topics; small $\beta$ favors topics concentrated on few words.

Inference: working backward from words to topics

Given only the observed words, LDA must infer the latent $\theta_d$, $\phi_k$, and per-word topic assignments $z$ — this posterior is intractable to compute exactly, so practical implementations use approximate inference: either collapsed Gibbs sampling (iteratively resampling each word's topic assignment conditioned on all others, converging to samples from the true posterior) or variational inference (optimizing a tractable approximate distribution to be close to the true posterior, faster but biased). Both converge to roughly the same qualitative topics on well-behaved corpora; Gibbs sampling is typically more accurate but slower, variational inference (the original LDA paper's method, and what most production libraries use by default) scales better.

lda_gensim.py

python

from gensim import corpora
from gensim.models import LdaModel
from gensim.utils import simple_preprocess

docs = [simple_preprocess(doc) for doc in raw_documents]
dictionary = corpora.Dictionary(docs)
dictionary.filter_extremes(no_below=5, no_above=0.5)  # drop rare/too-common terms
corpus = [dictionary.doc2bow(doc) for doc in docs]

lda = LdaModel(
    corpus=corpus,
    id2word=dictionary,
    num_topics=10,
    passes=20,
    alpha="auto",   # let the model learn document-topic sparsity
    eta="auto",     # let the model learn topic-word sparsity
    random_state=42,
)

for topic_id in range(10):
    print(f"Topic {topic_id}:", lda.print_topic(topic_id, topn=8))

# Infer a new document's topic mixture:
new_bow = dictionary.doc2bow(simple_preprocess(new_document))
print(lda.get_document_topics(new_bow))

Aspect	LDA	Modern alternative
Document representation	Mixture over K topics	Dense embedding (BERTopic uses transformer embeddings)
Interpretability	High — explicit word-topic distributions	Lower for raw embeddings; BERTopic recovers topic words via clustering
Handles short texts (tweets)	Poor — sparse word counts per doc	Better — embeddings capture meaning beyond raw counts
Choosing number of topics	Manual (coherence score, perplexity)	Can be semi-automatic (HDBSCAN cluster count in BERTopic)

Choosing K is more art than science

LDA requires specifying the number of topics $K$ upfront, and there's no single correct answer — too few topics merges distinct themes together, too many fragments a single theme across multiple nearly-identical topics. Practitioners typically sweep $K$ and select using topic coherence metrics (e.g., $C_v$ or UMass coherence, which measure how semantically related a topic's top words are) combined with human judgment on whether the resulting topics are actually interpretable and useful.

LDA topics are statistical co-occurrence patterns, not semantic categories

LDA has no understanding of meaning — a 'topic' is purely a learned pattern of which words statistically co-occur across documents. It can produce topics that are statistically coherent but semantically confusing (mixing genuinely distinct concepts that happen to co-occur in your specific corpus), and results are highly sensitive to preprocessing choices like stopword removal and vocabulary filtering.

What's Next?

LDA is a probabilistic, generative take on discovering latent structure in text; an earlier, purely linear-algebraic approach to the same underlying problem predates it by over a decade and is worth understanding on its own terms.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →