Difference-in-Differences Estimation
Why simple before/after comparisons mislead
Suppose a product team ships a new onboarding flow and retention goes up 5% afterward. Is the flow responsible? Maybe — or maybe retention was already trending up for seasonal reasons, or a concurrent marketing campaign drove the change. A naive before/after comparison on the treated group alone cannot separate the treatment effect from everything else that happened at the same time.
Difference-in-Differences (DiD) solves this by comparing the change over time in a treated group against the change over time in a comparable untreated (control) group. If both groups were drifting similarly before treatment, and only the treated group jumps afterward, the gap between the two changes is a credible estimate of the treatment effect — the confounding time trend cancels out.
The core estimator
Let $Y_{g,t}$ be the outcome for group $g \in \{treated, control\}$ at time $t \in \{pre, post\}$. The DiD estimate is:
$\hat{\tau}_{DiD} = (Y_{treated,post} - Y_{treated,pre}) - (Y_{control,post} - Y_{control,pre})$
Equivalently, this is the coefficient on an interaction term in a regression:
$Y_{i,t} = \beta_0 + \beta_1 \cdot Treated_i + \beta_2 \cdot Post_t + \beta_3 \cdot (Treated_i \times Post_t) + \epsilon_{i,t}$
Here $\beta_3$ is the DiD estimate: it isolates the extra change experienced by the treated group, net of the general time trend ($\beta_2$) and any fixed baseline difference between groups ($\beta_1$). Running it as a regression also makes it trivial to add covariates and cluster standard errors by unit, which matters enormously for correct inference.
The parallel trends assumption is the whole game
Choosing a control group
A good control group is exposed to the same macro shocks (seasonality, platform-wide changes, market conditions) as the treated group but did not receive the treatment. In a product rollout, this is often a geography or user segment where the feature wasn't shipped yet. In marketing measurement, it might be a region untouched by a specific campaign. The weaker the parallel-trends assumption looks, the more you should consider techniques like synthetic control (building a weighted combination of multiple control units that best matches the treated unit's pre-period trend) or matching on pre-period covariates before running DiD.
import numpy as np
import pandas as pd
import statsmodels.formula.api as smf
# Panel data: one row per unit-period, with group and post indicators
df = pd.DataFrame({
"unit_id": np.repeat(np.arange(200), 2),
"post": np.tile([0, 1], 200),
"treated": np.repeat(np.random.binomial(1, 0.5, 200), 2),
})
# Simulate outcome with a true treatment effect of 3.0
rng = np.random.default_rng(42)
trend = 1.5 * df["post"]
baseline_gap = 2.0 * df["treated"]
true_effect = 3.0 * (df["treated"] * df["post"])
noise = rng.normal(0, 1.0, len(df))
df["y"] = 10 + trend + baseline_gap + true_effect + noise
# The interaction coefficient is the DiD estimate
model = smf.ols("y ~ treated + post + treated:post", data=df).fit(
cov_type="cluster", cov_kwds={"groups": df["unit_id"]}
)
print(model.summary().tables[1])
# Look at the 'treated:post' row - its coefficient should land near 3.0
Clustering standard errors correctly
A common and costly mistake is computing standard errors as if each observation were independent. In panel data, outcomes for the same unit across periods are correlated, and ignoring this causes standard errors to be understated, sometimes drastically, which leads to false confidence in noisy effects. The fix is to cluster standard errors at the unit level (or at whatever level treatment was assigned — e.g. the store, the market, the user). With few clusters (under ~30-40), even clustered SEs can be unreliable, and you should consider a wild cluster bootstrap instead.
Extending to multiple periods and staggered adoption
Real rollouts rarely happen at a single clean moment — different units often adopt treatment at different times (staggered adoption). The classic two-way fixed effects regression generalizes DiD to this setting, but recent econometrics work (Goodman-Bacon 2021, Callaway and Sant'Anna 2021, Sun and Abraham 2021) has shown that naive staggered-adoption DiD can be badly biased when treatment effects vary over time or across cohorts, because earlier-treated units end up acting as flawed controls for later-treated ones. Modern practice uses estimators specifically designed for this — such as Callaway-Sant'Anna's group-time average treatment effects — rather than a single pooled interaction term.
DiD is a workhorse for causal inference without randomization
What's Next?
Difference-in-differences gives you an average treatment effect across a population, but it says nothing about which individuals benefited most. When you need to predict heterogeneous treatment effects at the individual level — to target interventions rather than just evaluate past ones — you move from this kind of population-level causal inference toward individual-level treatment effect estimation.
I build these systems professionally.
Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.