The peer review process: Rethinking by redoing

Many constituencies feel themselves ill-served by the peer review process, which constitutes a major determining factor in having an academic career.  The New York Times today looks at ways alternatives are starting to be constructed.  The issues raised are complex and important.  For example, it’s not clear that the changes envisaged would help all those disadvantaged by the current system.

However, let’s start at least  look at some of the remarks.  Here, for example, is a negative characterization of standard journal peer review:

The traditional method, in which independent experts evaluate a submission, often under a veil of anonymity, can take months, even years.

Clubby exclusiveness, sloppy editing and fraud have all marred peer review on occasion. Anonymity can help prevent personal bias, but it can also make reviewers less accountable; exclusiveness can help ensure quality control but can also narrow the range of feedback and participants.

In contrast:

 Open review more closely resembles Wikipedia behind the scenes, where anyone with an interest can post a comment. This open-door policy has made Wikipedia, on balance, a crucial reference resource.

But telling us that open review would make academic publishing more like Wikipedia is faint praise indeed. 

The most interesting part of the article, at least for me, concerns the varieties of alternatives.  Open review through journals, for example, appears still to let journals make major decisions about who gets reviewed.  But one might do something like Phil Papers, but with unpublished work.  Indeed, I don’t think Phil Papers is now restricted to published work, so perhaps it is a possible “open review” venue.

A major question is whether we will find analogues of current problems cropping up.  E.g., will we find women’s work of equal quality gets less attention? 

To end on a positive note:  the article notes that communicating is a deep academic value.  It is possible that a more open system could make publishing more like communicating and less like prize-winning.

3 thoughts on “The peer review process: Rethinking by redoing

  1. Hmm, a big topic – and a very good question about the share of attention that women’s, and other minorities’ work gets.

    1. The standard peer review may well have its faults. Its success, even from the traditional point of view, depends heavily on whether the people in the field are responsible and have integrity. In the end, the review process is only as good as the reviewers and editors are.
    (But it is important that there are fields – at least within my science, linguistics, there are definitely subfields like that – where the traditional peer review works pretty well.)

    2. The traditional peer review can be criticized for several reasons of very different nature. Sure, it allows a small educated circle to control the content of scientific publications, which in its turn ensures a certain quality of that content, validating it; but:

    2a. The circle may genuinely oppose new brilliant ideas which require it to rethink its own attitudes. (The “clubby”, “snobbish”, “intellectually inert” objection.)

    2b. The reviewing work takes a heavy toll on the members of the circle, and is not rewarded properly (sometimes, it is not rewarded at all.) (The “invisibility of the reviewer’s contribution” objection.)

    2c. The traditional review process does not use the full potential of the field, and because of that, is limited and consumes a lot of time. (The “ineffectiveness” objection.)

    Maybe there are more, but those are the main ones I can think of at the moment.

    So the real question is: how can we retain the good properties of the review while fighting some of the problems in 2a-2c?

    Simply rejecting any review process is possible – we just have to say: “from now on, we all publish stuff only in what is now called preprint archives”. The problem is, the volume of scientific content is gigantic, so people will have to make their choices what to read anyway. The only thing that will change will be that there will be no peer-review-sanctioned content, so people will have to use other information to find what interests them.
    And this, I suspect, will open the door for very serious discrimination. It is very hard to fight prejudice without having certain objective measures of equality. E.g., if you have no way to learn the amounts of pay different people receive, you have no way to prove pay discrimination, etc. Of course, the traditional peer review may involve discrimination, too; but then there are two stages: first, you can look at discrimination at the level of reviewers (e.g., by checking whether knowing the author’s gender influences the decisions); second, you can look at what happens after the reviewers make their decisions, taking those as “objective”. But if people simply read what they like, there is no process in between which you can try to check for fairness.

    But besides “everybody just reads what they like”, there are other alternatives. For instance, you can provide the infrastructure for other people than reviewers to add commentaries to scientific papers. If you do that after the traditional peer review process, you partly solve the problem 2c. If you do it at the same time with the traditional process, and the reviewers and editors may, but do not necessarily have to, use the open commentary, you might solve even more of 2c (but there’s also a risk that there won’t be enough interest in the community to do the open thing seriously: when it is a rare experiment, like the one described by NYT, it is one thing; when it is everyday practice for every article and book, well, you cannot really comment on any article and book, especially if they have not passed through the traditional review filter.) If you only use open commentary, you possibly solve the problem 2b as well, but at the risk of creating really snobbish new circles forming themselves.

    There are potential problems with such solutions leading to discrimination, but since the process is formal, you can confront them, just as you can fight the unfairness in traditional peer review.

  2. jj, do you have a link to a source that describes how the work of women and minorities ends up getting unequal attention? I think if we knew a little more about the reasons or the process by which this happens, then it might be easier to evaluate whether the open review would help, hinder, or remain similar to the status quo.

    There are a number of places where I can imagine bias entering the process. For example, the work of women and minorities could be rejected in the first sift by the editor without even sending it out to reviewers (which I believe is not a blinded process). This would seem to be unaffected by open review, if there is still a first sift and only some papers are put up for comment.

    But bias could come in at other places, too. For example if women were drawing on feminist theory and reviewers were not well-versed in feminist theory this could affect how “quality” is assessed. Open review might be able to aid in this kind of bias, assuming that the reviewers would be more diverse in an open method.

    I don’t have a link to a source, but I thought I read somewhere that when this was tried in the sciences it was not particularly effective because after some initial enthusiasm, it turned out that few people evaluated the papers. I believe the reason is similar to the “bystander effect” when a particular reviewer is charged with the review, then they feel responsible for the review. But when anyone is welcome to review, no one in particular is responsible for the review and so most people end up leaving it to someone else to do. I am not sure whether it is the case that the open review in the sciences fizzled out, but I seem to remember reading something about that.

  3. Hi,

    Thank you four your nice writing on The peer review process: Rethinking by redoing.

    Thanks.

Comments are closed.