BY AARON BARLOW
Retraction Watch, one of my all-time favorite blogs, posted an interview today with psychologist John Antonakis, who had identified the five diseases of academic publishing, significosis, neophilia, theorrhea, arigorium and disjunctivitis. I was immediately struck by the first:
Significosis is the incessant focus on producing statistically significant results, a well-known problem but one that still plagues us. Because the players in the publication game only consider statistically significant results as interesting and worthwhile to publish, the distribution of effect sizes is highly skewed. The potentially wrong estimates feed into meta-analyses and then inform policy. A result could be significant for many reasons, including chance or investigator bias, and not because the effect is true.
Significosis sidetracks all sorts of debates, including those over the situations of adjuncts and contingent hires in American colleges and universities. I’ve even seen people claim that adjuncts can make as much per hour as tenured faculty, so their situation must be pretty good. “Per hour,” for an academic position, is rather meaningless (corporatization has not gotten that far) and the question as to whether or not tenured faculty have it “good” is certainly debatable. The best way to understand adjuncts is to talk to them, many of them and at a variety of institutions. Trying to make numbers describe their experiences is dishonest in many ways, as even cursory familiarity with Darrell Huff’s old How to Lie with Statistics will remind us. Numbers have been in the driver’s seat in university settings for some time but they aren’t as foolproof as many would have us believe. In fact, they are often just the opposite. Significosis is a disease we all have to fight.
The other four diseases are equally as destructive to real intellectual discourse; all of them made me nod in recognition as I read the interview. We have all experienced each of them in our own scholarly pursuits, each of us contracting each from time to time. Even if we have recovered, we need to be vigilance against relapse.
The outright innumeracy of many scholars in the humanities is a far greater disease than the insistence on statistically significant results among the quantitatively inclined. The latter has its drawbacks, but they are modest compared to the benefit of keeping anti-empirical junk scholarship out of the journals. Take that away and you start to get articles on “feminist glaciation” or articles that make sweeping claims about the state of academia that are directly contradicted in empirical evidence.
You need to get out from under your numbers more, Phil. At least, don’t criticize the “sweeping claims” of others when you constantly make your own with no more (less, actually) justification. What you claim as “empirical evidence,” after all, is often no more than numbers.
Having empirical evidence for a claim is still firmer ground than simply making it up as you go. Or asserting something as “true” because you think it feels right, believe it confirms your political priors, or aligns with a non-randomly selected anecdote that you choose to prioritize.
But you ARE making it up as you go, Phil! Everything you argue, you assemble not for truth, but to confirm your prior belief. That’s why I never can take you seriously.
While Phil’s sweeping statements seem to be inflected by dubious positions the innumeracy of the humanities is a series problem. At least one of my colleagues did not understand how percentages work, most struggle with even basic numerical analysis, and one otherwise excellent senior academic simply skipped over tabulations (a point I had to explain to his PhD student). Asking them to understand statistical significance (and my experience almost none do) is too much of an ask.
Its notable that the quote presents statistical significance in a deeply misleading way and the subsequent discussion implies that you do not know or understand what it is or why it is used – there would be an irony in quoting someone complaining that the problem is a particular concept is not understood if indeed you do not understanding it yourself.
The quote is, of course, tongue-in-cheek, a satire.
I also am interested in what the term “significant” implies: it appears to me that a marginal or non-correlative finding may demonstrate significance as it eliminates an avenue of enquiry. Overall, academic findings will always be partially quantitative and partially qualitative: it is up to the author to demonstrate how these different measures are integrated into their findings.
Let’s hear you make that charge in a scholarly outlet then, Aaron! If you’re going to accuse me of misusing data in my published work – a very serious claim, I might add – then surely you can specify what I allegedly did wrong and submit it to peer review to see if it holds up.
Oh, please.
Always one for bold words…until it comes time to substantiate them.