Large pretraining corpora scraped from the web contain enormous amounts of near-duplicate and exact-duplicate text — mirrored pages, boilerplate licenses, quoted articles, scraper artifacts that hit the same URL multiple times, and auto-generated template pages that differ only in a product name or date. Left unaddressed, duplication isn't a harmless inefficiency; it actively degrades model quality. Models trained on heavily duplicated data have been shown to memorize duplicated passages disproportionately, waste compute re-learning the same tokens, and in some cases show measurably worse downstream performance than models trained on smaller, deduplicated datasets.

Exact vs. near duplicates

Deduplication at scale has to handle two distinct problems. Exact duplicates are byte-identical or near-byte-identical documents — these are caught by hashing the full document (e.g., with SHA-256 or MD5) and dropping repeats. Near duplicates are documents that share most of their content but differ slightly — a page re-published with a different header, a document with one paragraph edited, or boilerplate text wrapping different core content. Exact hashing is useless here because a single-character change produces a completely different hash. Near-duplicate detection requires similarity-based methods.

MinHash and Locality-Sensitive Hashing (LSH)

The standard technique for near-duplicate detection at web scale is MinHash combined with Locality-Sensitive Hashing. Each document is broken into overlapping n-grams (shingles), hashed with multiple hash functions, and represented by the minimum hash value under each function — the MinHash signature. The key property: the probability that two documents' MinHash signatures agree on a given hash function approximates the Jaccard similarity of their shingle sets, $J(A,B) = \frac{|A \cap B|}{|A \cup B|}$. LSH then buckets documents by bands of their MinHash signatures so that only documents likely to be near-duplicates are ever compared directly — making it feasible to find near-duplicates among billions of documents without an all-pairs comparison, which would be computationally impossible.

minhash_dedup.py

python

from datasketch import MinHash, MinHashLSH

def shingles(text, k=5):
    tokens = text.split()
    return set(tuple(tokens[i:i+k]) for i in range(len(tokens) - k + 1))

def minhash_signature(text, num_perm=128):
    m = MinHash(num_perm=num_perm)
    for shingle in shingles(text):
        m.update(" ".join(shingle).encode("utf-8"))
    return m

# Build an LSH index and flag near-duplicates
lsh = MinHashLSH(threshold=0.8, num_perm=128)
documents = {"doc1": "...", "doc2": "...", "doc3": "..."}

for doc_id, text in documents.items():
    sig = minhash_signature(text)
    duplicates = lsh.query(sig)       # existing docs similar enough to flag
    if duplicates:
        print(f"{doc_id} is a near-duplicate of {duplicates}")
    else:
        lsh.insert(doc_id, sig)

Suffix-array based exact substring deduplication

A complementary approach, used in pipelines like the one behind GPT-3's training set cleanup and formalized in work on 'deduplicating training data makes language models better,' builds a suffix array over the concatenated corpus to find exact-duplicate substrings of significant length (e.g., 50+ tokens) anywhere within or across documents — not just whole-document duplicates. This catches cases MinHash can miss, such as a long quoted passage repeated inside otherwise-different documents. Suffix arrays can be built in linear time and support this kind of substring-duplicate search efficiently even over terabyte-scale corpora.

Deduplication techniques compared

Technique	Catches	Scales to	Misses
Exact hashing	Byte-identical documents	Trivially, billions of docs	Any near-duplicate with edits
MinHash + LSH	Near-duplicate whole documents	Billions of docs with tuning	Short repeated substrings inside otherwise-different docs
Suffix array substring match	Exact repeated substrings anywhere	Terabyte-scale corpora	Paraphrased or reordered duplicates
Semantic embedding similarity	Paraphrases, translated duplicates	Expensive at full corpus scale	Requires an embedding model and vector search infra

Deduplication can over-correct

Aggressive deduplication thresholds can accidentally remove legitimately common but valuable content — canonical reference material, standard legal boilerplate that's informative in context, or frequently-cited passages that are repeated because they're important. Most production pipelines tune similarity thresholds carefully and deduplicate at multiple granularities (exact document, near-duplicate document, repeated substring) rather than applying one blunt global rule, and often sample a held-out set to sanity-check what's being dropped.

Deduplication is also a data-efficiency lever

Beyond quality, deduplication directly affects training economics: every duplicated token is compute spent re-learning something the model has already seen, with diminishing marginal benefit. For compute-constrained pretraining runs, a well-deduplicated dataset effectively increases the useful information density per token, letting the same compute budget buy more genuinely new signal.

What's next?

Deduplication is one piece of the broader discipline of curating pretraining data well — cleaning, filtering, and deduplicating raw web-scale text before it ever reaches a training run. The techniques here generalize beyond text too: similar hashing and nearest-neighbor ideas underpin deduplication for image and multimodal datasets as those corpora scale into the billions.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →