Defensive Gun Use in Politico Magazine: What to Believe

Recently, two authors--Evan DeFillipis and Devin Hughes--published an article in Politico Magazine criticizing a two-decade-old study by Gary Kleck and Marc Getz. Kleck just published a response. (The remainder of this post assumes you've read both.)
Let's look at what DeFillipis and Hughes said and what Kleck said in response. 

D & H: 

"In 1992, Gary Kleck and Marc Getz, criminologists at Florida State University, conducted a random digit-dial survey to establish the annual number of defensive gun uses in the United States. They surveyed 5,000 individuals, asking them if they had used a firearm in self-defense in the past year and, if so, for what reason and to what effect. Sixty-six incidences of defensive gun use were reported from the sample. The researchers then extrapolated their findings to the entire U.S. population, resulting in an estimate of between 1 million and 2.5 million defensive gun uses per year."

Kleck does not refute this. 

When presented with a list of reasons why his survey respondents might have had reason to exaggerate the number of times they used guns in self-defense--social desirability bias, awareness of the political context of the questions, and "telescoping"--Kleck does not refute any of these things, either. He doesn't say they don't matter; he doesn't argue against them, or even address them. 

His approach is, instead, to criticize DeFillipis and Hughes's credentials, say he's heard all of this before, question their motives, and pooh-pooh the very idea of criticism. 

His sole argument against DeFillipis and Hughes is this: "The authors’ discussion of possible flaws in survey estimates of DGU frequency is conspicuously one-sided, addressing only supposed flaws that could make the estimates too high—but none that could make the estimates too low."

In other words, 'It may be wrong one way, but it could be wrong the other way, too!'

He also makes numerous claims about supporting evidence, mentioning the "well-documented failure of many survey respondents to report criminal victimization, gun ownership or their own crimes" and saying "at least 18 national surveys have confirmed that DGUs are very common", but unlike DeFillipis and Hughes, he does not include specific results from other surveys that corroborate his beliefs, or include links to his supporting evidence. This is not persuasive. 

Finally, he postulates that DGUs would be underreported instead of overreported, because survey respondents could be admitting to criminal activity, and people are unlikely to report their own crimes. 

This seems to me to be nonsense, because Kleck and Geis conducted a presumably anonymous phone survey to get their results in the first place. They didn't ask people to march down to the local police precinct and confess. There would be no consequences for someone saying they used their gun in self-defense, even if it was illegal. 

Obviously, we're now getting into the heads of the survey respondents, and asking whether the biases DeFillipis and Hughes pointed out would have outweighed the tendency not to report a crime. But again, it's telling that DeFillipis and Hughes point to well-understood biases that have been confirmed by studies from other organizations. Kleck does not back up anything he says with links or the names of other organizations who have conducted studies to corroborate him. 

Ultimately, however, the question is whether a sample of 66 'yes' results out of 5,000 people can be fairly extrapolated to the whole country, what those unnamed eighteen studies Kleck said confirmed his results actually said, and what to make of the Arizona study, the NCVS data, and the Gun Violence Archive data in DeFillipis and Hughes's column that all provide reason not to accept Kleck's numbers at face value. 

On the face of it, there is much more reason to believe DeFillipis and Hughes--who seem to have made considerable efforts to survey the state of research in the field of defensive gun violence and account for biases--than to believe Kleck, who does not back up what he says with supporting evidence or other organizations, and who spends much more time bad-mouthing his critics than he does actually refuting what they have to say.