Weak Supervision Bootstrapped a Dataset We Couldn't Afford to Label by Hand

A client needed a dataset labeled for a document risk classification task at a volume — well into six figures of documents — that made full manual labeling a non-starter on the budget they had. Hand-labeling even a meaningful fraction at a reasonable per-label rate would have blown through the entire project budget before we touched model training. Weak supervision is what actually got this off the ground.

What weak supervision actually means here

Instead of asking humans to label every document, we wrote a set of heuristic labeling functions — simple rules based on keyword presence, document structure, metadata fields — each one individually noisy and each one covering only a subset of the documents. No single rule was reliable enough to trust on its own. Combined probabilistically, weighting each rule by its estimated accuracy against a small hand-labeled validation set, the aggregate label quality across the combined rules was good enough to train a usable initial model.

No single weak rule needs to be right most of the time. The combination just needs to be right more often than any individual rule, and that bar is much lower to clear.

Where the weak rules came from

We didn't invent the rules from nothing — they came from interviewing the subject matter experts who'd been manually assessing document risk before any of this, extracting the heuristics they already used intuitively and formalizing them into labeling functions. That grounding mattered: the rules weren't arbitrary guesses, they were an encoding of real domain expertise that just hadn't been captured in a form a model could learn from directly.

• Source labeling rules from actual domain expert heuristics, not invented guesses, for meaningfully better rule accuracy.
• Keep a small hand-labeled validation set regardless, to calibrate how much to trust each rule's output.
• Expect the first-pass weakly-labeled model to need a real manual-labeling refinement pass before production, not as a replacement for it.
• Weak supervision gets you a usable starting point fast — it is not a substitute for ground truth on your highest-stakes decisions.

The model trained on weakly supervised labels wasn't production-ready on its own — we used it to pre-filter which documents most needed human review, then spent the actual labeling budget on the smaller, higher-value subset the model flagged as uncertain. That combination is what made the whole project financially viable at all.