Pretraining a large language model on more data isn't automatically better — the composition and quality of that data shape model behavior as much as its sheer volume does. A trillion tokens of low-quality, repetitive, or narrow web text can produce a worse model than a smaller, carefully curated corpus. Dataset curation is the set of decisions and pipelines that determine exactly what text (and increasingly, what other modalities) a model actually learns from — source selection, quality filtering, deduplication, and domain/topic balancing.

Source selection and mixing

Modern pretraining corpora blend multiple sources deliberately rather than scraping the web indiscriminately: filtered Common Crawl web text, curated high-quality sources (Wikipedia, books, academic papers, code repositories), and increasingly, synthetic data generated by other models. The mixing ratio between these sources — what fraction of total training tokens comes from code versus web text versus books, for instance — is itself a major design decision with measurable downstream effects; upweighting code, for example, has been repeatedly observed to improve a model's general reasoning performance even on non-code tasks, not just coding benchmarks specifically.

Quality filtering

Raw web-scraped text contains large amounts of low-value content — boilerplate navigation text, spam, auto-generated pages, adult content, and text in languages or encodings the model doesn't need. Quality filtering pipelines typically combine: heuristic rules (minimum document length, perplexity-based filtering using a reference language model, repeated-n-gram detection to catch spammy or auto-generated text), classifier-based filtering (a trained model scoring documents for 'quality' against a reference set of known-good text, such as Wikipedia), and language identification to route or exclude content outside the target language set.

quality_filter_pipeline.py

python

def passes_quality_filters(document, reference_perplexity_model, quality_classifier):
    if len(document.split()) < 50:
        return False  # too short to carry much signal

    if has_excessive_repeated_ngrams(document, n=4, max_repeat_ratio=0.3):
        return False  # likely spam or auto-generated boilerplate

    perplexity = reference_perplexity_model.score(document)
    if perplexity > HIGH_PERPLEXITY_THRESHOLD:
        return False  # text a well-trained LM finds very "surprising" -- often garbled/low-quality

    quality_score = quality_classifier.predict(document)
    return quality_score > QUALITY_THRESHOLD

def has_excessive_repeated_ngrams(text, n, max_repeat_ratio):
    ngrams = [text[i:i+n] for i in range(len(text) - n)]
    unique_ratio = len(set(ngrams)) / max(len(ngrams), 1)
    return (1 - unique_ratio) > max_repeat_ratio

Filtering too aggressively narrows a model's knowledge, not just its noise

Quality classifiers are typically trained to prefer text resembling Wikipedia or edited prose — but aggressively filtering toward that style risks systematically excluding informal, dialectal, non-Western, or domain-specific text (forum discussions, non-standard English, technical jargon-heavy text) that doesn't match the reference 'quality' distribution but is still genuinely valuable training signal. Over-aggressive filtering trades some amount of noise reduction for reduced diversity and coverage, and this tradeoff is easy to get wrong silently, since its effects show up as subtle capability gaps rather than an obvious training failure.

Curation decisions and their primary effect

Curation lever	Primary effect	Risk if mishandled
Source mixing ratios	Shapes domain-specific capability (code, reasoning, factual recall)	Over/under-weighting a source skews capability profile
Heuristic quality filters	Removes boilerplate, spam, garbled text	Can incidentally strip valid informal/non-standard text
Classifier-based filtering	Finer-grained quality signal than heuristics alone	Inherits classifier's own training-data biases
Deduplication	Prevents overrepresentation of repeated content	Needs care not to also remove legitimately common, important text

Curation choices encode value judgments, not just technical quality judgments

Decisions about what counts as 'high quality' or 'relevant' text are not purely technical — they embed assumptions about whose writing and perspectives count as valuable training signal, which has direct downstream effects on whose knowledge, dialects, and viewpoints a model represents well versus poorly. Treating curation purely as a noise-reduction engineering problem, without examining what gets systematically excluded in the process, is a common and consequential blind spot.

What's Next?

Curation decisions interact closely with deduplication at scale — removing near-duplicate documents is itself a major curation lever with its own tradeoffs — and together these upstream data decisions set the ceiling on what later stages (pretraining itself, and all subsequent fine-tuning) can realistically achieve, no matter how well the training recipe downstream is tuned.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →