Synthetic Test Data Generation Saved Real QA Time, Here's the Setup

A CRM migration project needed thousands of realistic test records before we could validate the new field mappings, and the client's actual data was too sensitive to copy into a staging environment. The old approach — a QA engineer hand-crafting fifty edge cases — took two full days and still missed things.

We switched to generating synthetic records with an LLM, seeded with the real schema and a description of the weird real-world patterns we knew existed: international phone formats, names with diacritics, addresses missing a unit number, emails with plus-addressing. That seed list mattered more than the generation itself.

The setup that worked

Schema + 15 known edge-case patterns
   │
   ▼
LLM generates 2,000 synthetic records
   │
   ▼
Script validates: schema conformance, no duplicate PII patterns, distribution sanity checks
   │
   ▼
Human spot-checks 50 random records
   │
   ▼
Run against migration pipeline

The validation script step is the part most people skip, and it's the part that matters. The first batch the LLM generated had a subtle bias — almost every synthetic address was in a US format, even though we'd asked for international variety. Nobody would have caught that by eyeballing records one at a time, but a quick distribution check against the real production locale split flagged it immediately.

Synthetic data generation is only as good as the validation step you bolt onto it — the generation itself will confidently produce a biased sample and never tell you.

What changed

Two days of manual edge-case writing became about three hours: one to write the seed patterns, one to run and validate, one to spot-check and regenerate the biased batch. The migration itself caught four mapping bugs during QA that the old fifty-record manual set had never surfaced, simply because volume and variety matter more than cleverness when you're hunting for schema edge cases.

• Seed the generator with real known edge cases, don't rely on it to invent them from a blank schema.
• Always run a distribution sanity check against real production patterns before trusting the batch.
• Treat generated PII-shaped data as sensitive even though it's fake — don't ship it anywhere it could be mistaken for real.