Synthetic Data Generation Worked Better Than We Expected for a Cold-Start Problem

A client needed a classifier to route inbound leads by intent category, and the problem was simple to state and annoying to solve: they had no historical labeled data, because the categories themselves were new and nobody had been tagging leads that way before. Labeling enough real examples by hand to train anything useful would have taken weeks we didn't have before launch.

The stopgap that wasn't supposed to work this well

We generated a few thousand synthetic lead descriptions with an LLM, prompted with the category definitions and a handful of real examples the client could scrape together from memory, and trained a small classifier on that synthetic set as a placeholder while real labeled data accumulated. The plan was to swap the synthetic data out once we had a few weeks of real production labels. We never fully swapped it out, because the model trained on synthetic data was already performing within a few points of what the eventual real-data model achieved.

Synthetic data isn't a substitute for real data in general. It's a substitute for the specific real data you don't have yet, generated from the knowledge you do have about what that data should look like.

Why it worked here specifically

This wasn't a case where synthetic data papers over a fundamentally hard problem — the category definitions were clear, the distinguishing signal between categories was mostly lexical and structural rather than subtle, and the LLM generating the synthetic examples had plenty of relevant knowledge to draw on. Synthetic data generation does not rescue a problem where the real signal is something the generating model doesn't actually understand well.

• Category boundaries that are well-defined and explainable in a prompt generate better synthetic examples than vague or overlapping categories.
• Mix in whatever small amount of real data exists — even a few dozen real examples anchored the synthetic generation noticeably.
• Validate the synthetic-trained model against real production data as soon as any trickles in, don't wait for a full dataset.
• Watch for the model learning synthetic-data artifacts (overly clean phrasing, repeated sentence structures) rather than the actual signal.

We still replaced the synthetic data incrementally as real labels came in, and the model kept improving on the margins that mattered. But the fact that a synthetic-only model got a cold-start product to launch, with real accuracy numbers to show for it, changed how seriously we treat synthetic data as a first option now instead of a last resort.