Saturday, November 20, 2010

Posting reviews online does not affect review quality, but it does self-select a certain type of reviewer

Daphne Zaras forwarded me this article in BMJ (British Medical Journal): "Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial" (BMJ 2010; 341:c5729 doi: 10.1136/bmj.c5729).

Excerpted from the abstract:

Objectives To see whether telling peer reviewers that their signed reviews of original research papers might be posted on the BMJ’s website would affect the quality of their reviews.

Intervention Consecutive eligible papers were randomised either to have the reviewer’s signed report made available on the BMJ’s website alongside the published paper (intervention group) or to have the report made available only to the author—the BMJ’s normal procedure (control group). The intervention was the act of revealing to reviewers—after they had agreed to review but before they undertook their review—that their signed report might appear on the website.

Results 558 manuscripts were randomised, and 471 manuscripts remained after exclusions. Of the 1039 reviewers approached to take part in the study, 568 (55%) declined. Two editors’ evaluations of the quality of the peer review were obtained for all 471 manuscripts, with the corresponding author’s evaluation obtained for 453. There was no significant difference in review quality between the intervention and control groups (mean difference for editors 0.04, 95% CI −0.09 to 0.17; for authors 0.06, 95% CI −0.09 to 0.20). Any possible difference in favour of the control group was well below the level regarded as editorially significant. Reviewers in the intervention group took significantly longer to review (mean difference 25 minutes, 95% CI 3.0 to 47.0 minutes).

Conclusion Telling peer reviewers that their signed reviews might be available in the public domain on the BMJ’s website had no important effect on review quality. Although the possibility of posting reviews online was associated with a high refusal rate among potential peer reviewers and an increase in the amount of time taken to write a review, we believe that the ethical arguments in favour of open peer review more than outweigh these disadvantages.

Fifty-five percent declining is much higher than I've experienced in my editorship at the Electronic Journal of Severe Storms Meteorology (also an open peer-review journal at which I am Assistant Editor). I suspect that some didn't want to see their names associated with the paper as reviewers. Interestingly, we have seen that on occasion at EJSSM where reviewers decline when we say that we post signed reviews online.

That the quality of the reviews was not different whether or not the reviews were open was an interesting conclusion. This is perhaps not surprising. Of the reviewers who were willing to participate in this experiment, those who felt uncomfortable or insecure about posting their reviews online probably have a different personality type than those who don't care what the world or the authors thought of them. Therefore, the high decline rate helps self-select the right people who are willing to engage in this activity. This effect might explain why the reviews are not significantly different between the two groups.


Blogger El Gran Rogelio said...

Thanks for the post, Dave. That is a very interesting study, one that helps to answer some questions in a more defensible way when it comes to discussions of quality versus anonymity in reviews.

In the process of conceptualizing EJSSM's open-review policy, we (the founders) had no such studies to back us up on this point. Instead, it was more of a hunch, based on our collective experience as reviewers, editors and authors, that the open-review process wold yield high-quality reviews, even after the self-distillation of those who want to remain cloaked in anonymity. Fortunately, there are just enough idealists out there, and/or folks to just don't let themselves be bothered by their own identities, that EJSSM has benefited from numerous thorough, fair and rigorous reviews.

Ultimately, I see another big benefit to open review as immeasurable but enormously valuable: accountability. Reviewers aren't so likely to inject personal diatribes, "snark" comments, insults, windy tangents, or other impertinent and sometimes destructive rubbish into their reviews. Instead, the reviewer has good incentive to be honest yet fair about the manuscript under scrutiny, critiquing the science and not the author, going after the ideas and not the person, while still able to engage in all due rigor. This helps not just the authors, but the editors (makes our job easier...less policing to do!), and the reviewers themselves. As I explained to one initially hesitant reviewer who is a professor: You can do this review in your usual excellent manner, then show it to your students as an example of how to perform a scientific peer review.

===== Roger =====

December 27, 2010 at 1:36 AM  

Post a Comment

Subscribe to Post Comments [Atom]

<< Home