Latent Semantic Analysis
Finding meaning via linear algebra
Latent Semantic Analysis (LSA), also called Latent Semantic Indexing (LSI) in information retrieval, predates neural embeddings and topic models by decades (Deerwester et al., 1990). It discovers latent semantic structure in a document collection using nothing but matrix factorization — no probabilistic generative model, no neural network — applied to a term-document matrix.
The term-document matrix and SVD
Start with a term-document matrix $A$ of shape $|V| \times |D|$ (vocabulary size by number of documents), typically filled with TF-IDF weights. LSA applies Truncated Singular Value Decomposition to factor $A \approx U_k \Sigma_k V_k^T$, keeping only the top $k$ singular values (far fewer than the full vocabulary or document count — typically 100-300). $U_k$ gives a $k$-dimensional embedding for every term, $V_k$ gives a $k$-dimensional embedding for every document, and $\Sigma_k$ holds the singular values (relative importance of each latent dimension). This dimensionality reduction forces semantically related terms that never literally co-occur in the same document to still end up with similar vectors, because they're each correlated with the same underlying latent dimensions — this is how LSA captures synonymy (recognizing 'car' and 'automobile' as related) that raw TF-IDF cannot.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import Normalizer
vectorizer = TfidfVectorizer(stop_words="english", max_features=10000)
svd = TruncatedSVD(n_components=100, random_state=42)
normalizer = Normalizer(copy=False)
lsa_pipeline = make_pipeline(vectorizer, svd, normalizer)
document_vectors = lsa_pipeline.fit_transform(corpus) # shape: (n_docs, 100)
# Explore which original terms load most heavily onto a latent dimension
terms = vectorizer.get_feature_names_out()
component = svd.components_[0]
top_terms = [terms[i] for i in component.argsort()[::-1][:10]]
print("Top terms for latent dimension 0:", top_terms)
# Document similarity in the reduced semantic space
from sklearn.metrics.pairwise import cosine_similarity
sims = cosine_similarity(document_vectors[0:1], document_vectors)Explained variance and choosing k
Each singular value in $\Sigma$ represents the amount of variance in the original term-document matrix captured by its corresponding latent dimension, ordered from largest to smallest. Choosing $k$ trades off two failure modes: too small and you lose real distinctions between documents (over-compression, merging genuinely different topics); too large and you retain noise — idiosyncratic word usage that doesn't reflect genuine semantic structure, undermining the whole point of dimensionality reduction. A common heuristic is to plot cumulative explained variance ratio against $k$ and pick the elbow, similar to choosing components in standard PCA (LSA is, mathematically, PCA applied to a term-document matrix).
| Property | LSA | LDA | Neural embeddings (word2vec/BERT) |
|---|---|---|---|
| Underlying method | Matrix factorization (SVD) | Probabilistic generative model | Neural network trained on prediction objective |
| Interpretability of dimensions | Low — dimensions are abstract linear combinations | High — topics are word distributions | Low for individual dimensions |
| Handles polysemy | Poorly — single vector per term | Partially — word can belong to multiple topics | Contextual models (BERT) handle it well |
| Negative values in representation | Yes — SVD components can be negative | No — probabilities are non-negative | Yes |
LSA's negative weights are a real limitation
LSA is largely superseded but still useful as a fast baseline
What's Next?
Both LSA and LDA produce document representations meant to feed into practical applications — the most common of which is simply assigning documents to predefined categories.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.