Many constituencies feel themselves ill-served by the peer review process, which constitutes a major determining factor in having an academic career. The New York Times today looks at ways alternatives are starting to be constructed. The issues raised are complex and important. For example, it’s not clear that the changes envisaged would help all those disadvantaged by the current system.
However, let’s start at least look at some of the remarks. Here, for example, is a negative characterization of standard journal peer review:
The traditional method, in which independent experts evaluate a submission, often under a veil of anonymity, can take months, even years.
Clubby exclusiveness, sloppy editing and fraud have all marred peer review on occasion. Anonymity can help prevent personal bias, but it can also make reviewers less accountable; exclusiveness can help ensure quality control but can also narrow the range of feedback and participants.
In contrast:
Open review more closely resembles Wikipedia behind the scenes, where anyone with an interest can post a comment. This open-door policy has made Wikipedia, on balance, a crucial reference resource.
But telling us that open review would make academic publishing more like Wikipedia is faint praise indeed.
The most interesting part of the article, at least for me, concerns the varieties of alternatives. Open review through journals, for example, appears still to let journals make major decisions about who gets reviewed. But one might do something like Phil Papers, but with unpublished work. Indeed, I don’t think Phil Papers is now restricted to published work, so perhaps it is a possible “open review” venue.
A major question is whether we will find analogues of current problems cropping up. E.g., will we find women’s work of equal quality gets less attention?
To end on a positive note: the article notes that communicating is a deep academic value. It is possible that a more open system could make publishing more like communicating and less like prize-winning.