CRM Data Deduplication
Duplicates are a structural problem, not a data-entry accident
Every CRM with more than one path for data to enter — manual entry, form fills, CSV imports, integrations with other tools — will accumulate duplicate records, because none of those paths reliably check the full dataset for an existing match before creating a new one. Left unmanaged, duplicates don't just clutter reports: they split a single contact's engagement history across two records, so lead scoring undercounts their real activity, sales reps call the same person twice from different reps, and revenue reporting double-counts or misattributes a deal.
Exact matching vs. fuzzy matching
Deduplication logic operates on two tiers of matching confidence:
- Exact matching — comparing a normalized unique key, almost always email address for Contacts (lowercased, trimmed) or a normalized website domain for Companies. High confidence, low recall: it catches true duplicates reliably but misses anything where the key itself differs (a person using two email addresses, a company with multiple domains).
- Fuzzy matching — comparing multiple weaker signals (name similarity via string-distance algorithms, phone number, company name after stripping suffixes like 'Inc'/'LLC', postal address) and scoring the combined similarity. Higher recall, lower confidence: it catches duplicates exact matching misses, but also produces false positives (two different people named 'Michael Smith' at the same company) that need human review before merging.
Exact vs. fuzzy matching
| Dimension | Exact matching | Fuzzy matching |
|---|---|---|
| Matching key | Normalized email / domain | Name, phone, address, weighted similarity score |
| Confidence | High — rarely a false positive | Variable — needs a confidence threshold |
| Recall | Low — misses non-identical-key duplicates | Higher — catches near-matches exact logic misses |
| Safe to auto-merge? | Usually yes | Only above a high confidence threshold; else route to review |
Prevention beats cleanup: dedup at the point of entry
The cheapest duplicate to deal with is the one that's never created. Every integration, form handler, and import process should perform an upsert — search for an existing match against the defined key first, update it if found, create only if genuinely new — rather than a blind insert. This single discipline, applied consistently across every entry point, prevents the overwhelming majority of new duplicates; periodic cleanup jobs only have to handle the residual cases (legacy data, matching-key edge cases) rather than a constant new influx.
Comparing two Company records for a potential merge:
Record A: "Acme Corp" | acme.com | (212) 555-0100
Record B: "Acme Corporation" | acme.com | (212) 555-0199
Signal Match? Weight Score
Domain (normalized) exact 0.50 0.50
Company name (fuzzy) 0.88 sim 0.30 0.264
Phone (area+prefix match) partial 0.20 0.10
TOTAL: 0.864
Threshold: >= 0.85 -> auto-merge candidate (flagged for review)
0.60-0.84 -> manual review queue
< 0.60 -> not a duplicateMerge strategy matters as much as detection
MERGING Record A (primary, kept) + Record B (duplicate, merged away)
Field-level survivorship:
Email -> keep A if valid; else fall back to B
Phone -> keep whichever was most recently updated
Job Title -> keep whichever is non-blank; A wins ties
Lifecycle Stage -> keep the FURTHEST-ALONG stage of the two
Lead Score -> SUM both scores' behavioral components,
don't just keep one (loses real engagement)
Associated records:
All Deals, Tickets, and Activity history from B
-> re-associated to A, never deletedAuto-merging below a safe confidence threshold causes damage that's hard to reverse
What's next
Clean, deduplicated records are also the prerequisite for trustworthy pipeline metrics — a duplicated deal or contact distorts velocity and forecasting numbers just as badly as it distorts lead scoring.
Next: Pipeline Velocity Metrics →
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.