Peer Review: Make It Transparent

In 2012, I presented a paper at the Modern Language Association annual meeting that caused a small splash, especially for one line, “Blind peer review is dead. It just doesn’t know it yet.” There was a great deal of support for my position, but also quite a few who took umbrage. Most of these, I noticed, elided the word “blind” from their arguments, moaning that we have to have standards of some sort and that peer reviewers are the ones that uphold them.

It’s true, I have problems with peer review for reasons beyond the anonymity that often accompanies it, but it is the “blind” part that is most pernicious.

Two recent articles for Times Higher Education spotlight this and the wider problems once more, the more recent being a recounting of experiences with blind peer review and the older being a call for change.

The earlier article is by Richard Smith, former editor of the BMJ and chief executive of the BMJ Publishing Group from 1991 to 2004. He’s not some disgruntled assistant professor whose work isn’t seeing print but a major figure in medical scholarship. His words should carry weight (not that an assistant professor’s shouldn’t, but… ). He writes that “peer review persists because of vested interests. Absurdly, academic credit is measured by where people publish, holding back scientists from simply posting their studies online rather than publishing in journals. Publishers of science journals, both commercial and society, are making returns of up to 30 per cent and journals employ thousands of people.”

Peer review, in its current formulation, also exists because it provides a brake on innovation, something that scares people (though they would never admit it) at the top in just about every field. Successful innovation can break holds on power and prestige, as we’ve seen in media and journalism over the past decades, completely changing any particular landscape. Those who have already reached prominence in their professions want, at best, to be able to control change so that they can take advantage of it or, at least, not to be threatened by it. As Smith puts it, peer review remains, “ineffective, largely a lottery, anti-innovatory, slow, expensive, wasteful of scientific time, inefficient, easily abused, prone to bias, unable to detect fraud and irrelevant.” But it does serve a purpose, even if not one we should countenance.

Though I do think the day of blind peer review is over, progress to a replacement acceptable to appointment, tenure and promotion committees is slow. Not much has changed at all in the three-and-a-half years since my MLA panel presentation. The six contributions to the newer THE article, for the most part, present a convincing plaint against peer review as it still stands, dead or not. For example, “all referees will tell you that they are open-minded, write gently and are on the lookout for work of fabulous creativity. But they aren’t. It is hard for a human being to absorb ideas that are of first-order originality; such ideas, by definition, barely compute. Moreover, it is emotionally difficult for reviewers to be charitable about others’ manuscripts.” So true.

Another puts a finger on a new wrinkle: “Academics – and those academics transformed into ‘managers’ – have learned how to play around with peer reviewing and turn it to their advantage in determining who wins and loses in the local promotion stakes. And their use of peer review is used by open access publishers to excuse the venality of asking for money up front from authors (something that used to be derided as vanity publishing).”

A third writes, “In theory, peer review is a great idea, but as it generally operates, it sucks. The principal problem is the typical anonymity of reviewers. This can provide a cover for suppression of data that conflicts with the reviewer’s own results, sabotage of competitors’ funding applications, filching of research ideas or even the pursuit of petty vendettas.” Yup.

Another looks at the practical side, “the biggest problem is a lack of clarity of editorial decisions. Peer reviewers naturally vary in their rigour and style, but too many approach the process as a challenge to pick as many holes and to suggest as many new experiments as possible.” Another provides a warning, “Blind peer review should be fundamentally reformed. The lack of accountability is out of place when even job applicants can ask to see their references (and this was instituted to stop character assassination). As things stand, I even foresee some academics using Freedom of Information laws to flush out the identities of abusive reviewers.”

Personally, I like to see a variety of means to publication with varying editorial processes. This now exists, but professional evaluators in academia are still stuck in the peer-review mire, seeing the phrase as the most important academic imprimatur. What needs changing, as much as the blind peer review process itself, is how publications are considered for hiring, tenure and promotion.

Unfortunately, most committees are overworked and they look for shortcuts. The fact of something being peer reviewed makes it easier to accept without actually, say, reading the work in question. So we need to continue to push for change within the peer-review system, agitating for openness within the process, pointing to the many successful and open models that have appeared over the last decade. There is no reason that a reviewer should be anonymous. If one cannot stand behind one’s comments, one probably shouldn’t make them.

2 thoughts on “Peer Review: Make It Transparent

  1. Pingback: Weekend reads: Academic article brokering; favorite fieldwork bloopers; worst peer review ever - Retraction Watch at Retraction Watch

  2. It sounds from your post as if you are discussing single-blind peer review (the reviewer can see the authors, but the authors cannot see the reviewers). Most journals and nearly all the high quality conferences(*) in computer science work with double-blind peer review (the reviewer does not know who the authors are, and the authors do not know who the reviewers are), is this not the case in other fields?

    Anyway, I agree with several of your points, but I think that anonymity is important for reviewers. I’d rather see transparency in the form of reviews being published alongside the publication. Sure, it’s probably technically possible to de-anonymize reviewers with language analysis, but this is a small sacrifice for increased quality of reviews, and increased transparency of the peer review process.

    (*) these are usually the ones with two rounds of peer review, similar to journals, but many lower quality conferences also use double-blind reviews.

Your comments are welcome. They must be relevant to the topic at hand and must not contain advertisements, degrade others, or violate laws or considerations of privacy. We encourage the use of your real name, but do not prohibit pseudonyms as long as you don’t impersonate a real person.