A new post on Retraction Watch, “Peer review isn’t good at ‘dealing with exceptional or unconventional submissions,’ says study,” quotes the authors of the study of the title:
Because most new ideas tend to be bad ideas, resisting unconventional contributions may be a reasonable and efficient default instinct for evaluators. However, this is potentially problematic because unconventional work is often the source of major scientific breakthroughs.
This should be embroidered into samplers, etched onto the marble bases of statues, written by that old biplane in the sky above Coney Island, embossed on the covers of textbooks, sprayed as graffiti on the side of the George Washington Bridge and encoded as a Google banner ad.
We no longer know how to appropriately evaluate either learning or scholarship–or even art. If we ever did. We reduce it all to “outcomes” or “numerical assessment” or pass off responsibility to “peer review.” We only trust what we can define or count, or what comes from those we deem (often for rather arcane reasons) “experts.”
Evaluation, as it has evolved to the present day, is inherently regressive and constraining. It has no place for the unconventional or anything that challenges received knowledge.
In its post, Retraction Watch is discussing an article by Kyle Silera, Kirby Leeb, and Lisa Bero in Proceedings of the National Academy of Science, “Measuring the effectiveness of scientific gatekeeping.” The authors write:
Our research suggests that evaluative strategies that increase the mean quality of published science may also increase the risk of rejecting unconventional or outstanding work.
The same could be said of the standardized tests now so prevalent in American public schools and of the assessment paradigms being foisted on academic departments in colleges and universities. In all three cases, we are sacrificing the chance of the breakthrough, the surprising and the unconventional for incremental improvement–if that. The improvement lessens each cycle, the gatekeeping eventually resulting in stultification, “new” items becoming nothing more than the old presented slightly differently. A defined universe, after all, limits exploration within it simply through that definition–just as a Scantron test limits possible answers through a paucity of choice. We are creating a paint-by-numbers conception of the universe that leaves little real room for the really artistic or for the groundbreaking erasure, movement or addition of its lines and numbers.
What has happened to us?
Quite simply, we’ve grown fearful, especially fearful of failure. Worse, we’ve forgotten the value of failure, that it is not something to fear but something making real success possible. So, we’d rather be satisfied with the most constipated version of success possible than risk even the most minimal failure on the chance of creating something indubitably new. So, we want to create high-school graduates who are, for all intents and purposes, identical. So, we yearn for college classes stamped from molds. So, we cherish scholarship that challenges as little as possible.
We want to make sure, it seems, that our world changes as little as possible.
Not only is that a bore but, in current intellectual and physical climates, it is an invitation to disaster.