BY AARON BARLOW

“Peer review,” cried the provost, “that’s the gold standard.”

Sometimes it seems like the only standard. And we are being told to grasp it strongly.

Why not?

It lets us off the hook. Accepting it without question, we can ignore at least a couple of the urgent concerns regarding Reappointment, Promotion, Tenure (RPT) swirling about us today: In our exploded (and explosive) environment what is a “peer,” anyhow? And what does “review” mean when there are now myriad editorial structures and possibilities for evaluation? Come to think of it, does “peer review” mean the same thing in different fields? Should it?

Is “peer review” even appropriate as a single standard in fields as different as Physics and Culinary Arts?

When, for American scholars, recognition of peers eligible to conduct review was decided upon by colleagues who were editors housed at elite American or European universities (and who conveniently ignored the vast majority of otherwise qualified respondents at “lesser” institutions or elsewhere in the world), there was very little concern over selection of peers as reviewers. Both reviewers and editors were part of a gatekeeping structure of long-standing tradition.

Today, a peer reviewer can be almost anyone. There are no standards and the editor/gatekeepers come from diverse environments, some of them outside of academia. The decisions they make are as wide-ranging as are, today, the putatively “academic” journals and presses appearing in every field. So great has been the growth of scholarly venues contributing legitimately to every conceivable discipline that we can no longer even limit ourselves to what we once considered “top tier” venues, for the changing streams of publication have swept many of them into backwaters.

The “review” of “peer review” has always been suspect. For generations, scholars have complained about reviewers who used their positions to further their own positions and students and to black-ball those of their competitors. Long has been the struggle for openness in peer review, a move away from the “blind peer review” that was once the academic standard. In some fields, though, review prior to publication has been—and is—critical and should not be constrained.

One of the first questions of reviewer in a scientific field is, “Are the results replicable?” That is, would a repetition of the events of the study lead to the same conclusions? In some fields, this has fallen away as a part off pre-publication review to the extent that, years later, articles are having to be retracted for the simple reason that others cannot reach the same results. This is unfortunate.

In other fields, though, replicability is irrelevant. How would it apply to Victorian literature? Oh, maybe someone providing a statistical analysis of word usage in Wilkie Collins might expect their numbers to be challengeable, at least, but, that type of study is still an outlier. Perhaps awareness of prior work (“review of the literature”) needs to be shown, but that’s not even always the case. So, just what should a reviewer be looking for?

Different fields are addressing this question in differing ways, some moving to open processes of review that themselves can become part of the intellectual discussion necessary for keeping all academic activity alive. Others are experimenting with different alternatives, many possibly just as positive. Unfortunately, the range of review possibilities, even in restrictive fields, shows little cohesion, making any discussion of the meaning of “review” almost—meaningless.

Our contemporary over-reliance on peer review as the gold standard in RPT stems, in part, from the fact that our departmental peer committees are often overwhelmed and underprepared when it comes to the decisions brought before them. As it has grown into an emerging global network, academia has also fractured, pieces within particular disciplines becoming unrecognizable (and unevaluatable) to people within those very fields. Thus, unable to personally evaluate even the work of departmental colleagues, peers have turned to reliance on the reviewers selected by the venues of publication.

This reliance begs two questions: Are the reviewers up to the task? And, Are the specific review processes appropriate to the particulars of any one field?

As more and more institutions of higher education are emphasizing, more and more strongly, faculty contributions to scholarship as part of the RPT process, it is becoming more and more important to move beyond a reliance on the simplistic vision of peer review that may have once sufficed. It is time for provosts and the faculties they lead to begin to develop RPT standards and procedures appropriate to the individual needs of their institutions and the divergent ones of their departments. At the same time, we need to start investigating the methodologies of review—and gatekeeping—that surround us.