Added, from Jenny Saul: “Those who want more will want to look at Carole Lee and Christian Shunn’s paper on philosophy review practices. A key point that comes out there is how much nastier philosophers are than other reviewers studied.”
In the last couple years, I have presided over or assisted in peer-review processes for journal issues, anthologies, and conferences in Philosophy, with one consistently repeated shock across all venues, at least in my limited experience so far: It seemed to me as if anonymized peer-review seemed to bring out something vindictive in almost half of referees. Everyone who’s had an infamous “Reviewer #2” experience may be nodding right now, but I did not expect this. (I’ve gotten my own wee share of mean reviews, yes. But I am still surprised.) It caused me to seriously question whether doubly anonymous peer review is proven to be effective and good. I also thought that perhaps my impression was idiosyncratic.
I went looking for research to reaffirm the worth of peer-review, but I found little empirical verification that peer-review in journals achieves desired ends. I was relieved to find Hilda Bastian’s recent PLOS blog post, “Weighing Up Anonymity and Openness in Publication Peer Review,” in which she announced she had “taken a deep dive into this literature.” She only makes three unqualified statements, and the first of them is there is not a lot of great data:
But first, what evidence do we have that masking the identities of authors and peer reviewers achieves what it is meant to?
Well, it’s complicated. Which means it really needs a solid, up-to-date systematic review… We don’t have an overwhelming evidence basis for anything.
Ouch. That gets me right in the justified true belief. Her second firm finding confirmed something I’ve always longed to resist when students and colleagues allege it, just because it’s rather depressing (condensed below to avoid Bastian’s penchant for referring to anonymity as “blind”):
Institutionalizing anonymity [is] only partially successful at hiding authors’ identities, and mostly only when people in their field don’t know what authors have been working on.
Admittedly, she focuses on biomedical publications, but her review of the evidence includes non-biomed pubs, notably Budden’s (2008) comparative study suggesting that Behavioral Ecology saw more women published after changing to doubly anonymized peer review (which we have previously posted on here). She does not find that this study compellingly establishes that anonymizing authors reduces gender bias, although she notes evidence that at some science journals, “odds are stacked against women,” and there are “clear signs of other biases that have been shown at some journals,” notably status bias.
The only other really conclusive finding she offers is one that underlines the problem which sent me on my hunt:
On the other hand, the anonymity of peer review reports definitely enables negative, and even egregious, behavior.
Take heart, those of you with Reviewer #2 scars! You are not alone. Peer reviewers were more likely to be courteous when they, the reviewers, did not have anonymity:
Peer reviewers were more likely to substantiate the points they made when they knew they would be named. They were especially likely to provide extra substantiation if they were recommending an article be rejected, and they knew their report would be published if the article was accepted anyway.
In some studies, when the reviewers knew they would be named, they were likely to be more courteous or regarded as helpful by the authors.
There’s no support here for the concern that naming peer reviewers leads to systematically less critical reviews – and some support for improvement.
There was one large effect: many peer reviewers declined the invitation to peer review when they knew there was a chance they would be named – especially when they knew their colors would be nailed to the public mast if the article was published.
The results of Bastian’s investigations give me some hope that it is possible to gather evidence helpful to imagining better systems of quality-control and publication. I remain committed to anonymizing authors, since status bias seems no better to me than gender bias. But the porousness of author-identity masking, and the conduct of anonymized referees, gives me food for future thought.
Feminist Philosophers 