# Pre-Testing in a DiD Setup using the did Package

## Introduction

This vignette provides a discussion of how to conduct pre-tests in DiD setups using the did package

• One appealing feature of many DiD applications with multiple periods is that the researcher can pre-test the parallel trends assumptions.

• The idea here is simple: although one can not always test whether parallel trends itself holds, one can check if it holds in periods before treated units actually become treated.

• Importantly, this is just a pre-test; it is different from an actual test. Whether or not the parallel trends assumption holds in pre-treatment periods does not actually tell you if it holds in the current period (and this is when you need it to hold!). It is certainly possible for the identifying assumptions to hold in previous periods but not hold in current periods; it is also possible for identifying assumptions to be violated in previous periods but for them to hold in current periods. That being said, we view the pre-test as a piece of evidence on the credibility of the DiD design in a particular application.

• In this vignette, we demonstrate that the approach used in the did package for pre-testing may work substantially better than the more common “event study regression”.

## Common Approaches to Pre-Testing in Applications

By far the most common approach to pre-testing in applications is to run an event-study regression. Here, the idea is to run a regression that includes leads and lags of the treatment dummy variable such as $Y_{it} = \theta_t + \eta_i + \sum_{l=-\mathcal{T}}^{\mathcal{T}-1} D_{it}^l \mu_l + v_{it}$

where $$D_{it}^l = 1$$ if individual $$i$$ has been exposed to the treatment for $$l$$ periods in period $$t$$, and $$D_{it}^l = 0$$ otherwise. To be clear here, it is helpful to give some examples. Suppose individual $$i$$ becomes treated in period 3. Then,

• $$D_{it}^0 = 1$$ when $$t=3$$ and is equal to 0 in other time periods

• $$D_{it}^2 = 1$$ when $$t=5$$ and is equal to 0 in other time periods

• $$D_{it}^{-2} = 1$$ when $$t=1$$ and is equal to 0 in other time periods.

And $$\mu_l$$ is interpreted as the effect of treatment for different lengths of exposure to the treatment. Typically, $$\mu_{-1}$$ is normalized to be equal to 0, and we follow that convention here. It is common to interpret estimated $$\mu_l$$’s with $$l < 0$$ as a way to pre-test the parallel trends assumption.

### Pitfalls with Event Study Regressions

#### Pitfall: Selective Treatment Timing

Sun and Abraham (2021) point out a major limitation of event study regressions: when there is selective treatment timing the $$\mu_l$$ end up being weighted averages of treatment effects across different lengths of exposures.

Selective treatment timing means that individuals in different groups experience systematically different effects of participating in the treatment from individuals in other groups. For example, there would be selective treatment timing if individuals choose to be treated in earlier periods if they tend to experience larger benefits from participating in the treatment. This sort of selective treatment timing is likely to be present in many applications in economics / policy evaluation.

Contrary to event study regressions, pre-tests based on group-time average treatment effects (or based on group-time average treatment effects that are aggregated into an event study plot) are still valid even in the presence of selective treatment timing.

To see this in action, let’s keep the same example as before, but add selective treatment timing.

In contrast to the last case, it is clear that things have gone wrong here. Parallel trends holds in all time periods and for all groups here, but the event study regression incorrectly rejects that parallel trends holds – this is due to the selective treatment timing.

We can compare this to what happens using the did package:

This is the correct performance (up to aforementioned caveats about multiple hypothesis testing).

## Conditional Moment Tests

Another main use case for the did package is when the parallel trends assumptions holds after conditioning on some covariates. This is likely to be important in many applications. For example, to evaluate the effect of participating in a job training program on earnings, it is likely to be important to condition on an individual’s education. This would be true if (i) the distribution of education is different for individuals that participate in job training relative to those that don’t (this is very likely to hold as people that participate in job training tend to have less education than those who do not), and (ii) if the path of earnings (absent participating in job training) depends on an individual’s education. See Heckman, Ichimura, and Todd (1998) and Abadie (2005) for more discussion.

Even when one includes covariates to estimate group-time average treatment effects, pre-tests based only on group-time average treatment effects can fail to detect some violations of the parallel trends assumption. To give an example, suppose that the only covariate is binary variable for an individual’s sex. Pre-tests based on group-time average treatment effects could fail to detect violations of the conditional parallel trends assumption in cases where it is violated in one direction for men and in the other direction for women.

The did package contains an additional pre-test for the conditional parallel trends assumption in the conditional_did_pretest function.

# not run (this code can be substantially slower)
reset.sim()
set.seed(1814)
nt <- 1000
nu <- 1000
cdp <- conditional_did_pretest("Y", "period", "id", "G", xformla=~X, data=data)
cdp