Choosing the right comparison

Almost every policy finding depends on a comparison. Black voters had a rejection rate of 2.1%—compared to what? Higher than before SB 202—but by how much? Choosing the right comparison group and the right baseline period is as important as the data itself. A poorly chosen comparison can make a non-effect look like a disparity, or hide a real disparity.

Baseline selection

A baseline is the reference point against which you measure change. For voting rights analysis, the most common baselines are: (1) the same jurisdiction in a prior election cycle (before-and-after comparison); (2) a different jurisdiction in the same cycle (cross-sectional comparison); or (3) a theoretical standard like the statewide average.

Before-and-after baselines are powerful when the policy changed between the two periods—that's what makes the comparison informative. Cross-sectional comparisons are useful when policy varies across jurisdictions.

Comparison group pitfalls

Non-equivalent comparison groups: Comparing Fulton County (urban, 8 million voters) to Echols County (rural, 4,000 voters) as if they are equivalent ignores systematic differences that explain most of the gap. Any comparison must ask: are the groups similar on other relevant dimensions?

Cherry-picking the comparison period: Choosing a baseline period that maximizes the apparent disparity—without disclosing that choice—is a common analytical integrity problem. Show your baseline selection decision and note its effect.

Changing denominators: Comparing "mail ballots rejected" (a count) in 2022 to "mail ballot rejection rate" (a rate) in 2020 is not a valid comparison. Counts and rates measure different things.

What makes a valid counterfactual

A counterfactual is a statement about what would have happened under different conditions. In policy analysis, the strongest counterfactuals are those supported by comparable data. The ensemble simulation described in Module 3 is a form of counterfactual: "What would the map look like if neutral criteria had been applied?" The comparison is between the enacted map and the distribution of neutral maps.

For mail ballot analysis: "What would the rejection rate be if all counties applied the same signature-matching standard?" That counterfactual is not directly observable, but you can approximate it by analyzing counties that explicitly adopted a more permissive standard and comparing their rejection rates to those that did not.