Making referee reports more transparent

From the very interesting new blog,   New APPS: Art, Politics, Philosophy, Science

Here’s a simple proposal (due to Alva Noe, as far as I can tell): accepted, refereed papers and books should be accompanied with the name of the referees and, ideally, their reports, if only, in the online edition. (Note rejections can still be done anonymously.)

The author, Eric Schliesser, sees a number of benefits, including the improvement in referee diligence and the transparency of connections between editors, ingroups, etc.

Read the full post here.

15 thoughts on “Making referee reports more transparent

  1. What a perfectly foul idea. Referees are writing to the author, with suggestions for how to improve the manuscript. It’s to be hoped the author takes the suggestions, duly improves the manuscript, and thanks the anonymous referees nicely in the acknowledgments. How embarrassing for the author to have all the weaknesses or downright errors in the penultimate draft pointed out for all the world to see!

  2. Hilde Lindemann, you are right! I am embarrassed to say that somehow I just skipped over the part about the reports being published. The name of the referee I’d favor, but certainly not the report – or so I now think.

  3. I have to disagree with Prof. Lindemann, on two counts:
    (1) The potential for embarrassment seems to me like a rather small risk. In other contexts we either don’t worry much about embarrassment or think that we should worry less about it. We think we should call out sexist behavior, for example, even if doing so is embarrassing to the individual who perpetrated that behavior. Maybe there are other considerations (viz., justice) in this sort of case that are missing in the case of journal submission reports, but that just means it’s not obvious that embarrassment is a sufficient reason not to make the reports publicly available.
    (2) The potential for embarrassment here seems no worse than the potential for embarrassment at conference talks, after lectures, in seminars, &c. A devastating objection or a glaring error can be pointed out just as easily in public venues as in the report. Indeed, if anything, the pressure to respond immediately to the devastating objection or the glaring error would make the potential for embarrassment worse in these situations.

    That said, I don’t think we (whoever `we’ refers to) should immediately do away with anonymous peer review all together. Perhaps a couple of journals could experiment with a system where authors have the option to have the reports on their submissions made available publicly.

  4. Um, but Dan, this is not a good analogy. Hilde’s whole point was that by the time the paper comes out you have presumably responded to the criticisms of the report, so this is publicizing criticisms of an old, now-non-existent draft.

    I am confused by the whole proposal. If we don’t publish the reports – and I agree with Hilde that we shouldn’t – then what is supposed to be the big advantage from publishing the names? I don’t get the plus side.

    I will say – if I knew my name was going to be revealed to a paper-writer if the paper gets accepted it would *seriously* curtail my willingness to give honest criticisms. Just because a paper is overall worth publishing doesn’t mean the submitted draft doesn’t have real problems of the sort that I might not want to mention without anonymity, especially since as the reviewer I still wouldn’t know who the author is and hence whom I might be insulting or annoying. Plus sometimes I write a negative report on a paper and it gets accepted anyhow … then what? This proposal seems to me to be all pain and no gain. What problem is it supposed to be addressing?

  5. I’m inclined to think revealing the names makes writing a report more like being a critic at a conference. Given some of the utterly irresponsible things I’ve seen in referee reports, that seems to be a good idea.

    It’s also the case that the editor’s selection of referees can be hugely important; it would be telling if, e.g., we see the gool ol boys are passing on the gool ol boys.

  6. It’s hard enough for editors to get people to agree to be referees and to get them to write reports in a timely manner. This would make both things much harder, further slowing down the process. It’s a really terrible idea!

  7. Anonymous: I’m reluctant to use “its too hard and will delay things” at this stage. Eventually, people will do what it is the custom to do, and the stages on the way to becoming a custom may vary.

    One thing that might be tried is making the referee identity optional for a while.

    I am not in favor of publishing the reports. I have to spend a lot of time polishing work to be published, and reports do not need the perfect metaphor, still less perfect spelling.

  8. So now, on this plan, when I write a report I have to worry that I might be panning the paper of some powerful person in the field whose paper will get accepted anyhow (let’s face facts) and s/he will know it was I who panned? And untenured folks and grad students have to put themselves in this position too? No thanks! I think this would seriously hamper the honesty of the reviewing process.

  9. I think I’m with Rebecca on this. There could also be revenge rejections if people knew the identity of referees.

  10. The sort of fairness and transparency that Alva Noe is after is surely best achieved with proper anonymous review?

  11. I think Rebecca’s remark sets up the issue in the wrong way: no review should involve ‘panning’ a paper. If the remarks are critical and show what is wrong with the paper, then reasonable authors will be grateful.

    Ok, ok that assumes that authors are reasonable.
    And I agree that there are worries about those in vulnerable positions losing out.

    But it does seem to me that it could make for greater accountability at journals, though: paper gets published by Big Shot Prof, and it isn’t great – as is flagged up in the publicly available Reviewers remarks. Then a) Reviewer gets credit for calling out the problems, and b) Editor has harder time justifying the decision to publish not great paper.

    In the case that Hilde Lindemann highlights, where the review refers to the problems in a previous draft – agreed, no one wants errors in their first drafts publicly paraded. But on the other hand, isn’t it useful and fair for reviewers to be given credit for (what may be substantial) suggested changes?

    Hmm. I’m undecided about this matter – I can see pros and cons.

  12. Stoat, agreed. I’m glad that people have presented some important cons; I called the APPs attention to these, and think they should be part of future discussions.

    I think also that non-tenured scholars should be given the protection of anonymity, whatever the result.

  13. I’m with Monkey in being with Rebecca on this one. The cons of removing anonymity seem significant. If one of the problems is in finding willing, competent, and responsible referees, it is not a good idea to create disincentives to people volunteering and offering an honest assessment. I do a fair amount of refereeing and I wouldn’t feel comfortable assessing papers if I knew that my verdict was made public.

    I said this on the other blog, but one legit worry has to do with the way in which editors pair papers with referees. We want to avoid situations where people accept papers written by chums, for example. One way to cut down on this is at the back end by having referees names released to the public, but another way would be to have people monitoring the editors. Seems it would have a similar effect. We also could just institute rules that require no institutional connection between refs and authors. That could be done without removing anonymity. If we all include information about employment history and educational history, editors can be required to find refs that don’t overlap with authors. That could help and would be very easy to institute.

  14. Rebecca is right. In fact, I suggest a truly anonymous reviewing process in which not even the editor knows who submitted the paper–the name can be revealed iff the paper gets accepted. Why suppose that the editor is not biased by knowing that the person submitting the paper is (female) prof. X from (unknown) university Y? And probably Sartre is right that when you choose your advisors, you’ve often already chosen what to do.

  15. here’s a .pdf of a nature article about molecular biologists publishing referee reports (w author replies) online. might be useful to know how it’s gone for them. (tho, NB, their submissions, in the first place, are ‘single blind’; that is, the authors are not anonymous.)

    Click to access embo-transparency.pdf

Comments are closed.