Estimating the effect of concealed carry laws on murder: A response to Bondy, et al

Carlisle Moody
Economics Department
College of William and Mary
Williamsburg, VA 23187-8795
[email protected]

and

John R. Lott
Crime Prevention Research Center
[email protected]

August 1, 2023
Introduction
In 2021 we wrote a short paper (Lott and Moody 2022) to illustrate a concern arising from the
use of a single dummy variable to characterize the effect of right-to-carry laws on murder and
other violent crimes. The problem arises when researchers apply two-way fixed effects (TWFE)
models to more recent data, truncating the sample. This truncation can be justified in a variety of
ways. For example, according to Siegel, et.al. (2017, p. 1924),

Moreover, Ayres and Donohue (2003) and Hepburn et al. (2004) have suggested that the
relationship between concealed-carry laws and homicide rates may have been different
during the period before and after the early 1990s. In addition, studies that included
homicide rates from before 1994 were examining a trend that was increasing, whereas
studies examining homicide rates after 1994 were capturing declining trends. For these
reasons, a reexamination of this research question with more recent data is needed.

Using the sample 1991-2018, they find that right-to-carry laws significantly increase homicide
and firearm homicide.

Another example is Zimmerman (2014), which is concerned primarily with the effect of private
security on crime. Zimmerman includes a dummy variable to control for the effect of right-to-
carry laws. Because of limitations in the availability of private security data, Zimmerman limits
his study to 1999-2010. He finds (p.71),

The shall-issue coefficient takes a positive sign in all regressions save for the rape model
and is statistically significant in the murder, robbery, assault, burglary, and larceny
models. These latter findings may imply that the passage of shall-issue laws increases the
propensity for crime….

Finally, Donohue (2017) examines panel data from 1991-2015. He finds that, “right-to-carry
laws increase total, firearm, and handgun homicides, while showing no statistically significant
effect on nonfirearm homicides and long-gun homicides.” (p. 1864)

It is self-evident that the effect of right-to-carry laws, for good or ill, will increase with the
number of concealed carry permits. The problem is that right-to-carry laws vary dramatically
across states, and there are systematic differences in the laws between early and late-adopting
states. The more costly it is to obtain a permit the fewer people will carry. The late-adopting
states, such as Illinois, were reluctantly dragged into the right-to-carry column. These states tend
to have much higher fees and longer training requirements and have adopted more restrictive
regulations on using the permits. For example, in Illinois, the last state to adopt right-to-carry
laws, the average permit fee is $150, about three times the average cost of getting permission to
carry across states (Lott, 2022). In addition, only one other state has a training requirement of 16
hours that by itself can cost $250 to $300. By contrast, 32 states allow people to carry without a
mandatory training requirement. Twenty-seven “constitutional carry” states now have no fees or
training requirements for people to carry.

1
While the number of permits have increased in all states over the last few decades, the
percentage of the adult population with permits increased much less in the later adopting states.
Thus, the effect of the law can be expected to be greater in the earlier adopting states. Why then
do studies using only late-adopting states show a significantly positive effect when the growth
rate of concealed carry permits is lower, due to the fees and restrictions imposed, than in states
that adopted the law earlier? It seems unlikely that the effects (positive or negative) would be
larger in late-adopting states with fewer permits per capita.

We posited the “novel theory” that the resulting coefficient could be biased upward, perhaps
even becoming positive and significant, because it compares later adopting right-to-carry states
with those that adopted the law earlier. Obviously, if 11 states adopted right-to-carry laws prior
to 1991, and the sample begins in 1991, then all 11 will be consigned to the omitted class since
they have not switched in the sample period. Therefore, the coefficient on the right-to-carry
dummy will necessarily be comparing the effect in the late-adopting states to the average effect
in the 11 early adopting states as well as the never-adopting “may-issue” states. If the effect of
right-to-carry laws on crime is negative and early adopters reduce crime more than late adopters,
then the effect could be estimated as positive.

To account for these differences, we separated the states into early and late adopters using 1991
as the separator, following Siegel (2017) and Donohue (2017). There was also the problem of
states that had switched from right-to-carry to permit-less or constitutional carry (CC) laws,
which have no fees or training mandates, making it even easier for people to carry handguns and
thus presumably have an even bigger increase in the percentage of adults who carry concealed
firearms. We accounted for these switchers by adding a dummy for constitutional carry laws as a
control variable. We then set out to demonstrate this hypothesis with a simple test.

First, we estimated the data on the whole sample (1970-2018), starting before Alabama passed
their right-to-carry law in 1975. We included a single right-to-carry dummy and a single
constitutional carry dummy, and other standard control variables. For those states that switched
from right-to-carry to constitutional carry, we kept the right-to-carry dummy equal to one
because both laws were in effect at that time. The result was that both the coefficients on the
right-to-carry and constitutional carry dummies were negative.

Then we created two dummies one for early adopters and one for late adopters (leaving the
never-adopters as the excluded class) and estimated that regression. The coefficients on both the
early and late dummies were negative. Therefore, the late adopters also experienced lower crime
rates after the passage of right-to-carry laws. Now we attempted to answer the question, what
happens when we truncate the sample, using only the sample beginning in 1991? Answer: the
coefficient on the late adopters became positive. The only thing we did was truncate the data. We
did not change the model. The coefficient switched to positive, not because the law increases
crime in right-to-carry states, but that late adopting states have smaller increases in permits and
thus reduce crime by less than early states. We thought this result worthy of publication.

2
After we published our article, we became aware of an expanding literature in difference in
differences (DID) analyses that further developed the concerns we raised. These studies criticize
two-way fixed effects analyses like ours (and Siegel et al. (2017), Donohue (2017), and
Zimmerman (2014)). For example, "Recent research has shown that… TWFE estimators are
unbiased for an ATE [average treatment effect] if parallel trends holds, and if another
assumption is satisfied: the treatment effect should be constant, between groups and over time.
Unlike parallel trends, this assumption is unlikely to hold, even approximately, in most of the
applications where TWFE regressions have been used." (deChasemartin & D'Haultfoueill 2022,
p.2, henceforth CD.)

Indeed, the problem frequently arises, according to CD, when researchers make "forbidden
comparisons" defined as comparisons of treated groups with already treated groups. The results
can be so biased that "… one could have that [the TWFE coefficient] is negative while all the
ATEs are positive. (CD 2020, p. 2965) The scales fell from our eyes. Right-to-carry researchers
were comparing the late-treated states with early, already-treated states, the exemplar of
forbidden comparisons that do, as our paper shows, lead to sign changes. Our simple example
was an illustration of a recently discovered econometric problem with TWFE policy analysis.

The requirement for a valid comparison is that the parallel trends assumption must hold, and the
treatment must be constant across treated groups and over time. We assumed, along with every
right-to-carry policy analysis of which we are aware, that the treated states had parallel trends.
We also assumed that the effect of the right-to-carry laws were constant across states, an
assumption every right-to-carry study that uses a single dummy makes. Finally, the assumption
of time homogeneity is also required to make valid comparisons in difference-in-differences
studies. Our paper is an example of what might happen if the treatment effect is not constant
across states and over time.

A year after our study (Lott and Moody 2022) was published. Bondy, et. al (2023) published a
critique. Their critique misses the point of our article completely. They did not address the
problem of invalid comparisons for right-to-carry studies that use truncated samples at all.
Instead, they focused their attention on some fragility tests of our paper. These fragility tests are
irrelevant, as we argue below. Perhaps their most potentially damaging criticism is that the
parallel trends assumption may not be accurate for our study. They present a figure representing
an event study using right-to-carry and constitutional carry dummies and claim, "The pre-
treatment values are often dramatically different from zero and exhibit highly unstable values
over time." (Bondy, et. al. 2023, p. 4) Yet they do not present an F-test on the joint significance
of the pre-event placebo dummies. Why not? Possibly because the F-test is insignificant, judging
from their 95% confidence intervals. We address the rest of their criticisms by re-estimating our
model using the methods of deChaismartin and D'Haultfoueill (2020), which avoid forbidden
comparisons and are robust with respect to time and group heterogeneity.

3
We also note that our sample was truncated to 2014 because we did not have observations for the
Donahue-Levitt abortion variable for 2015-2018. 1 However, we have since discovered that it is
easy to extrapolate the time series since the variable is a smooth trend. We tested the
extrapolation method by setting the last six years of the abortion variable for murder aside and
extrapolating the data from 2008 to 2014. We computed the resulting estimates' mean absolute
percentage error. The mean absolute percentage error for the effective abortion rate for murder
and violent crime is 0.6 percent. The following graph shows the resulting data for the Donohue-
Levitt effective abortion rate for murder and our extrapolation for Illinois from 2000 to 2018.
The results for other states are similar.

Figure 1

For this study, we use essentially the same set of control variables we used in the original study,
namely the Donohue-Levitt abortion variable, prison incarceration per capita (lagged), sworn
police officers per capita (also lagged), the unemployment rate, construction employment per
capita (the construction industry concentrates large numbers of young men), the proportion of the
population between 15-39, the proportion of black males 15-39, population density, and the
Fryer et al (2013) measure of cocaine (to control for the crack epidemic). Aside from the
extrapolated values of the abortion variable, the only change is that we now combine the separate
beer, wine, and spirits into a single measure of alcohol per capita. We include the constitutional
carry dummy as a control in the right-to-carry analysis and vice-versa. Finally, we also include a
lagged dependent variable to capture dynamic effects and a set of year dummies. We now have
complete data from 1970-2018.

1
See also Lott and Whitley (2007).

4
Using the entire sample, we re-estimate our model using two-way fixed effects. The results are
presented in Table 1.

Table 1: Right-to-Carry law, effect on the murder rate

(1) (2) (3) (4)
1970-2018 1991-2018
VARIABLES Coef T-ratio Coef T-ratio

Right to Carry -2.937* -1.802 0.926 0.435

Constitutional carry -4.615 -1.335 -3.033 -0.847
Lag of incarceration per capita -2.290*** -3.109 -3.395** -2.389
Lag of police officers per capita -2.763 -1.308 1.105 0.344
Effective abortion rate -10.54*** -8.219 -13.65*** -6.247
Unemployment rate -0.378 -0.925 0.871 1.226
Construction employment per capita 0.446*** 3.829 0.772*** 3.929
Alcohol per capita -0.0979*** -4.334 -0.182*** -4.710
Crack cocaine 0.792 1.128 -0.104 -0.0768
Population density 0.112*** 3.972 0.269*** 4.676
Percent population 15-39 1.090** 2.159 -0.242 -0.218
Percent black males 15-39 5.688** 2.458 15.85*** 3.603
Yt-1 0.390*** 21.12 0.280*** 10.45
N 2,449 1,400
*** p<0.01, ** p<0.05, * p<0.1

The results in columns 1 and 2 indicate that both the right-to-carry and constitutional carry
dummy variables have negative coefficients, although neither is significantly different from zero
at the .05 level (although the right-to-carry dummy is significant at the .10 level). We cannot
help but note that the Donohue and Levitt abortion variable is negative and significant, indicating
that murder studies that do not include it must suffer from omitted variable bias. Yet Donohue,
the co-discoverer of this potentially relevant variable, never uses it in his right-to-carry analyses.
One wonders why.

The next step is to determine whether the estimated coefficients are biased by negative weights
generated by the invalid comparison of treated states with those already treated (early adopters).
Thanks to deChaismartin and D'Haultfoueill, STATA now discovers such negative weights and
alerts the unwary TWFE analyst to invalid comparisons. We applied their program to our TWFE
model in Table 1. The estimate of the right-to-carry coefficient is a weighted sum of 865
estimated average treatment effects on the treated (ATT), of which 66 (7.6%) have negative
weights. According to (2020, p. 2966), “If many weights are negative and the ratio is not very
large…” they recommend doing their difference-in-differences analysis which is robust to
heterogeneous treatment effects. Since the ratio is small and insignificant, and we have no
threshold to determine if 66 negative weights are too many, we conclude that bias is possible.

We also estimated the same regression model but limited the sample to 1991-2018. The results
are presented in the third and fourth columns of Table 1. The coefficient on the right-to-carry

5
dummy is now positive but not significant. According to our negative weight analysis, there are
now 608 ATT’s receiving positive weights and 209 (!) ATT’s with negative weights.
Apparently, limiting the sample to late-adopters causes the number of forbidden comparisons to
increase and biases the TWFE coefficient. Who knew?

Because of the potential bias of our TWFE coefficients, we estimate the corresponding event
studies using the deChaisemartin and D’Haultfoueill (2020) estimator. An event study using the
entire sample from 1970 to 2018 using the Stata program did_multiplegt generates the results
shown in Table 2 and Figure 2 with respect to the effect of right to carry laws on murder. We test
whether placebo laws passed in the four years leading up to the implementation of the policy are
significantly different from zero using a standard F-test. If the coefficients on the pre-event
placebo laws are not significantly different from zero as a group, we are comfortable assuming
parallel trends in the treatment period. The placebo law F-statistic for the right-to-carry analysis
was not significantly different from zero with a p-value of 0.99, indicating that the parallel trends
assumption is justified.

Table 2: Effect of Right-to-Carry and Constitutional Carry laws on murder

Right to carry Constitutional carry
Estimate SE T-ratio Estimate SE T-ratio
Year_0 4.30 4.00 1.07 8.42 6.69 1.26
Year_1 5.89 3.77 1.56 -4.60 8.23 -0.56
Year_2 7.33 5.15 1.42 -10.96 9.02 -1.22
Year_3 1.97 6.51 0.30 -14.62 7.61 -1.92
Year_4 0.58 7.14 0.08 -18.17 23.73 -0.77
Year_5 -2.03 7.05 -0.29 -35.26 22.61 -1.56
Year_6 -5.81 9.32 -0.62 -21.11 20.93 -1.01
Year_7 -3.84 7.61 -0.50
Year_8 -15.74 9.61 -1.64
Year_9 -14.76 9.48 -1.56
Year_10 -12.75 9.23 -1.38
Year_11 -8.57 10.42 -0.82
Year_12 -11.98 12.01 -1.00
Average -3.96 6.00 -0.66 -7.42 7.52 -0.99

6
Figure 2

Overall, the effect of the right to carry law on murder is negative but not significantly different
from zero. After a brief increase, the effect on the murder rate becomes permanently negative.

7
Figure 3

Since the joint placebo test (p=.07) does not reject the null hypothesis of parallel trends at the .05
level and the coefficients for the three years closest to the event are virtually on the zero line, we
are comfortable assuming parallel trends. We do not include as many dynamic effects as in the
right to carry graph since only three states, Alaska, Arizona, and Wyoming, have had
constitutional carry laws for over three years. The remaining constitutional carry states in the
sample are as follows (number of years since passage in parentheses): Kansas (3), Mississippi
(3), Idaho (2), West Virginia (2), Missouri (1), and North Dakota (1). We are assuming that the
experience of the later adopters will be similar to the early adopters. Although not significant, the
overall effect is negative after the year of adoption.

Perhaps, in this new age of difference-in-differences estimation, the only relevant criticism from
Bondy et al. is that their application of the deChaisemartin-D’Haultfoueill (2020) procedure
yields a positive effect of both right-to-carry and constitutional Carry laws on the murder rate.

We note that we applied the De Chaisemartin and d’Haultfoeuille (2020) estimator to

Moody and Lott’s Model (1) and found a positive [italics in original] but not statistically
significant estimate of the effect of shall-issue laws on 100 times the log homicide rate of
2.10 with a standard error of 8.18. For permitless carry, we find an even greater point
estimate of the increase in homicide of 8.76. This demonstrates that heterogeneous
treatment effects across adopting states significantly bias Moody and Lott’s results.
(Bondy et al 2023, p.7)

8
How could this happen? It turns out that Bondy, et. al. achieved this result by doing a one-year
event study, reporting only the results for the year of adoption. Bondy, et. al do not provide an
explanation as to why they chose to do the event study this way. We cannot think of a
justification, given the obvious effect of time on the number of concealed carry permits. Clearly,
they do not do this in their TWFE models where the right-to-carry dummy takes the unit value
all the way through the end of the sample. Because year zero is the year of implementation and
many of these laws take effect in the middle of the year, there is likely to be a small effect. In
addition, states usually issue very few permits immediately after implementing right-to-carry
laws perhaps because the issuing agency is not fully prepared or is understaffed, etc. And,
obviously, the effects could change as more people acquire right-to-carry permits or begin
carrying their firearms without permits in a constitutional carry state.

Using our 1970-2018 sample and following Bondy, et. al, we did a one-year event study. The
results were identical to the Year_0 results in Table 2. The initial year zero effect is independent
of the number of years in the event study. If Bondy, et. al had done a dynamic event study over a
reasonable period, they would have discovered the overall negative effect of both the right-to-
carry and constitutional carry laws as the number of permits or permit-less carrying increases.

Conclusion

The Bondy, et. al criticisms inaccurately describe what we did and provide selective and
misleading results. More importantly, they completely missed the point of our analysis, namely
that applying TWFE to a truncated sample, say 1991-2018, causes the resulting coefficient to be
biased by invalid comparisons of newly treated states to previously treated states. Even if the
actual right-to-carry dummy is negative, the estimated coefficient could be positive. These biased
results can be ameliorated by using the new DID estimators, which are robust to time and state
heterogeneity, but it is always better to use a complete sample which has all the information
available.
The Bondy, et. al two-way fixed-effects models are old-fashioned and irrelevant. Using all the
data and event analyses that ignore forbidden comparisons and are robust to time and group
heterogeneity, we find evidence that both right-to-carry laws and constitutional carry laws reduce
murder in the long run as the number of permits or permit-less carrying increases. Although the
standard errors indicate that these coefficients are not precisely estimated, it cannot be claimed
that either one of these laws significantly increases murder. Complete results, programs, and data
are available at <link>.

9
References
Ayres I and Donohue J.J. 2003. Shooting down the “more guns, less crime” hypothesis. Stanford
Law Review 55:1193–1300.
Bondy, M., Cai, S., and Donohue, J.J. 2023. Estimating the Effect of U.S. Concealed Carry Laws
on Homicide: A Replication and Sensitivity Analysis. International Review of Law and
Economics 75. https://doi.org/10.1016/j.irle.2023.106141

deChaisemartin, C. and D’Haultfoueille, X. 2020. Two-way fixed effects estimators with

heterogeneous treatment effects. American Economic Review 110(9): 2964-96.

_____. 2022. Two-way fixed-effects and difference-in-differences with heterogeneous treatment

effects: A survey. https://ssrn.com/abstract=3980758.
Donohue, J.J. 2017. Laws Facilitating Gun Carrying and Homicide. American Journal of Public
Health 107(12): 1864–1865.
Donohue, J.J. and Levitt, S. 2020. The impact of legalized abortion on crime over the last two
decades. American Law and Economics Review 22(2): 241-302.
Fryer, R.G., Heaton, P.S., Levitt, S.D., Murphy, K.M. 2013. Measuring Crack Cocaine and Its
Impact. Economic Inquiry 51(3) :1651-1681.
Hepburn L, Miller M, Azrael D, Hemenway D. 2004. The effect of nondiscretionary concealed
weapon carrying laws on homicide. Journal of Trauma. 56(3): 676–681.

Lott, J.R. and Moody, C.E. 2022. Do Right to Carry Laws Still Reduce Violent Crime?
Academia Letters, https://doi.org/10.20935/AL4888.

Lott, J.R. and Whitley, J. 2007. “Abortion and Crime: Unwanted Children and Out-of-Wedlock
Births,” Economic Inquiry 45(2): 304-324.

Lott, J.R. “Concealed Carry Permit Holders Across the United States: 2022,” Crime Prevention
Research Center Working paper,
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4279137.

Siegel, M., Xuan, Z., Ross, C.S., Galea, S., Kalesan, B., Fleegler, E., Goss, K.A. 2017. Easiness
of access to concealed carry permits and homicide rates in the United States. American
Journal of Public Health 107(12): 1923-1929.

Zimmerman, P.R. 2014. The deterrence of crime through private security efforts: Theory and
evidence. International Review of Law and Economics 37: 66-75.

10