A CRM Deduplication Project That Took Four Times Longer Than Scoped

The scope on paper was simple: merge duplicate contact and company records in a CRM with roughly 200,000 contacts. I estimated three weeks based on a sample of a few hundred records that looked like clean, obvious duplicates — same email, slightly different name formatting. The project took twelve weeks, and the reason is that the sample I scoped from wasn't representative of the actual mess.

Duplicates aren't random, they're evidence

Once we got into the full dataset, a pattern emerged: duplicates clustered heavily around specific integrations. A marketing automation sync that created a new contact every time a form was submitted without a lookup against existing email, a sales tool that created company records from email domains without checking for an existing match, and a data enrichment vendor that occasionally returned a slightly different company name for the same domain, which the CRM treated as a new entity. Every cluster of duplicates was a symptom of a specific upstream integration doing the wrong thing, not a random data quality issue.

A deduplication project that only merges records and doesn't fix the integration creating new duplicates is a subscription, not a fix.

What actually ate the extra nine weeks

• Building merge logic that correctly chose the 'surviving' record's fields without silently dropping sales notes or activity history attached to the record being merged away.
• Finding and fixing the three integrations actively creating new duplicates daily — without this, the cleanup would have been undone within a month.
• Handling the long tail of fuzzy-match duplicates (typos, abbreviations, old company names after acquisitions) that automated matching flagged as uncertain and needed human review.
• Re-running the merge in batches with rollback checkpoints after the first full-dataset run accidentally merged two genuinely different companies with similar names.

The lesson I actually apply now: never scope a dedup project off a sample without first segmenting duplicates by likely source. A sample of clean, obvious duplicates will always underestimate the project, because the hard duplicates are hard precisely because they're produced by a process that's still running and actively creating more of them while you clean up the backlog.