Cleaning Roster Datafor Union Campaign Success
Module 5 · Stripping Formatting and Deduplicating 5.4 Fuzzy duplicate detection
Subsection 5.4

Fuzzy duplicate detection

~8 min

Worked example · Fuzzy matching

The hardest duplicate case is when the rows differ in ways the normalizer cannot fix:

  • Maria Vasquez
  • Maria Elena Vasquez (with a middle name)
  • M. Vasquez (initial instead of first name)

The normalizer turns these into maria vasquez, maria elena vasquez, and m vasquez — still three different strings.

Step 1 · Anchor on the last name + ZIP

Most fuzzy duplicates share two things: the last name and the address. Build a second helper column that concatenates last name and ZIP:

=LOWER(TRIM(B2)) & "|" & TEXT(H2, "00000")

This makes vasquez|30310. All three Marias above produce the same key.

Step 2 · Count appearances on the new key

Run =COUNTIF(O:O, O2) on the new key column. Any value above 1 is a candidate fuzzy duplicate.

Step 3 · Pick the source of truth

For each duplicate set, pick the row with the most filled fields as the source of truth. Add a Field_Count column that counts non-blank fields per row:

=COUNTA(A2:L2)

Sort the duplicate set by Field_Count descending. The top row is the source of truth. Mark it green. Mark the others red. Decide whether to merge fields from the red rows into the green row.

Action: Mark this page complete when you have finished the activity above.