Document Similarity Measures
How similar are these two pieces of text?
Document similarity underlies search ranking, deduplication, recommendation, plagiarism detection, and clustering. There's no single 'correct' similarity measure — the right choice depends on whether you care about exact lexical overlap, semantic meaning, or structural/syntactic resemblance, and each notion of similarity requires a different representation and distance function.
Lexical similarity: measuring literal overlap
Jaccard similarity treats each document as a set of tokens (or n-grams) and measures overlap: $J(A, B) = \frac{|A \cap B|}{|A \cup B|}$. It's simple, fast, and effective for near-duplicate detection (two documents differing only slightly in wording will have high Jaccard similarity on shared n-grams) but ignores word frequency and order entirely. Cosine similarity over TF-IDF vectors improves on this by weighting shared terms by their informativeness: $\cos(\theta) = \frac{\vec{A} \cdot \vec{B}}{\|\vec{A}\| \|\vec{B}\|}$. Both methods fundamentally require literal shared vocabulary — they'll score 'the cat sat' and 'the feline rested' as dissimilar despite near-identical meaning.
Semantic similarity: measuring shared meaning
Embedding-based cosine similarity addresses the vocabulary mismatch problem by first encoding each document into a dense vector using a sentence embedding model (sentence-transformers, or pooled/mean-pooled transformer representations), then computing cosine similarity between those vectors. Because the embedding space is trained to place semantically similar text close together regardless of exact wording, 'the cat sat' and 'the feline rested' end up with meaningfully high similarity. The quality of this approach is entirely bounded by the embedding model's training objective and domain coverage — a general-purpose embedding model may underperform on highly technical or jargon-heavy domains it wasn't trained on.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity as sklearn_cosine
from sentence_transformers import SentenceTransformer, util
doc_a = "The quick brown fox jumps over the lazy dog."
doc_b = "A fast, dark-colored fox leaps above a sleepy canine."
# --- Lexical similarity: nearly zero overlap in exact wording ---
vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform([doc_a, doc_b])
lexical_sim = sklearn_cosine(tfidf_matrix[0], tfidf_matrix[1])[0][0]
print(f"Lexical (TF-IDF) similarity: {lexical_sim:.3f}") # likely low
# --- Semantic similarity: captures the paraphrase relationship ---
encoder = SentenceTransformer("all-mpnet-base-v2")
emb_a, emb_b = encoder.encode([doc_a, doc_b])
semantic_sim = util.cos_sim(emb_a, emb_b).item()
print(f"Semantic (embedding) similarity: {semantic_sim:.3f}") # likely high| Method | Captures paraphrase | Compute cost | Best use case |
|---|---|---|---|
| Jaccard (set overlap) | No | Very low | Exact near-duplicate detection |
| TF-IDF cosine | No | Low | Lexical search, fast baselines |
| Sentence embedding cosine | Yes | Moderate (needs encoder inference) | Semantic search, deduplication by meaning |
| Cross-encoder (joint scoring) | Yes, most accurately | High (no precomputed index possible) | Reranking a small candidate set |
Bi-encoders vs. cross-encoders is the key production tradeoff
Similarity scores from different models aren't comparable
What's Next?
A particularly high-stakes application of document similarity — detecting when two texts express the same meaning in different words — deserves its own dedicated treatment given how much it matters for deduplication and content integrity systems.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.