There are way more manuscripts than opportunities for respected peer-reviewed publications (Sinhababu 2016). So many good manuscripts might never be properly reviewed (or published). This would be bad. In this post, I’ll mention a few potential solutions. Then I’ll briefly evaluate one: eliminating compulsory peer-review altogether.
PEER-REVIEW IS A RECENT THING
I learned from Kate Norlock that peer-review is a relatively recent thing.†
… the surprisingly short history of what we now think of as peer-review [Times Higher Ed.] … the Google ngram on peer-review: [Google ngram article] …. suggests that academics have only been so fixated on it as the measure of our worth since the 1970s.
CURRENT PEER-REVIEW NORMS ARE NOT OBVIOUSLY THE BEST
One reason for peer-review might be that it inhibits bias. And there is some evidence that anonymous peer-review reduces bias (Budden et al 2008). However, a review of 17 studies challenges this finding (Bastian 2015; for more discussion see Norlock 2015).††
Perhaps further reviews of the evidence would undermine our other reasons for existing peer-review practices — feel free to point me to such reviews in the comments. Whatever the evidence is, if it does not obviously favor the status quo, then we should investigate our alternatives.
ALTERNATIVES TO THE STATUS QUO
As Kate Norlock (and Aileen Fyfe) mentioned, the current norms of peer-review are relatively new. And as Hilda Bastian explains, the apparent reasons for current peer-review practices are not well supported by evidence. So let’s consider some alternatives to current peer-reviewed publication practices. Starting with the most drastic…
- No more journals, publishers, or top-down quality control, ever. That is, we publish stuff whenever we want, wherever we want, however we want (repositories, personal websites, etc.).
- Keep journals and other publication schemes, but no more top-down quality control. Journals, publishers, etc. would be for organizational purposes only — e.g., Ethics continues to publish stuff about ethics.
- Keep journals and other publication schemes, and keep editors, but no more referees. Journals, publishers, etc. would serve organizational and minimal quality control functions — e.g., editors could still desk reject papers that do not meet basic academic standards, but they would publish everything else.
- Keep journals and other publication schemes, keep editors, and keep referees, but don’t base publication decisions on referee reports. Referees still complete reports, but editors do not have access to them until after a decision has been made. Maybe editors could append (signed?) reviewer comments (ratings?) to every published paper (van Rooyen, Delamothe, and Evans 2010).
- Open peer review. Keep journals and other publication schemes, keep editors, and keep referees, but make reviewers sign their referee reports (van Rooyen 1999, Walsh et al 2000).
- Whatever we do, make publication less of a priority. For instance, we could assign less weight to publications in decisions to hire, promote, fund, invite to [whatever], etc.
- Whatever we do, stop attributing prestige to certain journals and publishers. Maybe just stop ranking journals altogether …or at least stop ranking them badly (see “Academic Tech: Custom Reporting & Ranking“)
(There are other possibilities that I have ignored; feel free to add to this list in the comments. )
ELIMINATING PEER-REVIEW ALTOGETHER
Our first possibility is tantamount to eliminating compulsory peer-review altogether. We can easily imagine potential pros and cons of this proposal.
- The opportunity to prioritize other stuff. Imagine how much time and energy would be freed up by eliminating compulsory review! Academics would not be bogged down by manuscript reviews, editors would not be bogged down by the search for reviewers, etc. Who knows, maybe this time could be better spent on something else.
- Faster publication cycle. One of the primary reasons for delayed publication is peer-review. It takes time to find reviewers, wait for their reports, and make a decisions based on reviewers’ reports. Without a compulsory review process, this delay is eliminated. Research could be available much faster. This would have benefits for both academics and whoever consumes academic research.
Regarding the first point, it’s not entirely clear whether the value of other stuff is higher than the value of compulsory peer-review. What we need are credible analyses of the costs and benefits of each.
Something similar can be said about the second point: We need to know more about the costs and benefits of both a faster publication cycle and compulsory peer-review. Until we know more, it’s not clear whether the benefit(s) of faster publication would be a net benefit.
- Less dialogue. If we eliminate compulsory peer-review, then there will be less dialogue between academics. This might result in — among other things — academics work being less inclusive, interdisciplinary, rigorous, etc.
- Fewer indicators of quality. Without the stamp of approval from prestigious peer-reviewed journals, book publishers, conferences, etc., it will be harder to separate the seemingly good stuff from the seemingly not so good stuff.†††
- More errors. With fewer people checking our work, there might be more errors in publications.
The concern about dialogue might not be warranted. There are other mechanisms for dialogue besides peer-review. Indeed, academics engage in other forms of dialogue without being compelled to do so. For example, academics are often invited (not on the basis of peer-review) to present and answer questions about their research. So there would still be opportunities for dialogue without compulsory peer-review. The question is whether the value of compulsory peer-review can be provided by these other forms of dialogue. Further, the claim that this would cause less dialogue is a testable hypothesis. I do not know of good evidence that supports the hypothesis.
The concern about indicators of quality is probably unwarranted. After all, academics can determine quality without compulsory peer-review. For instance, academics can (and do) look to the wisdom of the crowd (of readers). And they can (and do) rely on their own judgments. So it is not clear what academics would lose in the way of quality indicators if compulsory peer-review were eliminated.
I am not sure what to make of the concern about errors. After all, additional people working on a manuscript can easily result in more errors, not less. What do the data say about the average quantity (and severity) of errors that are prevented by compulsory peer-review?
The publication of good research is delayed and even prevented by the resource-intensive demands of compulsory peer-review. One potential fix is the elimination of compulsory peer-review. In order to evaluate this proposal, we would need to determine the costs and benefits of compulsory peer-review. We’d also need to determine the costs and benefits of the alternatives.
Bastian, H. (2015, May 13). Weighing Up Anonymity And Openness In Publication Peer Review. [Free]
Budden, A. E., Tregenza, T., Aarssen, L. W., Koricheva, J., Leimu, R., & Lortie, C. J. (2008). Double-blind review favours increased representation of female authors. Trends in Ecology & Evolution, 23(1), 4–6. [Free] [Paywall]
Norlock, K. (2015, July 7). Anonymity of peer review reports “definitely” enables egregious behavior. [Free]
Sinhababu, N. (2016, June 7). 2,000 Spaces for 10,000 Papers: Why Everything Gets Rejected & Referees Are Exhausted [Free]
van Rooyen, S., Delamothe, T., & Evans, S. J. W. (2010). Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. BMJ, 341, c5729. [Free]
van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999). Effect of open peer review on quality of reviews and on reviewers’recommendations: a randomised trial. BMJ, 318(7175), 23–27. [Free]
Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000). Open peer review: a randomised controlled trial. The British Journal of Psychiatry, 176(1), 47–51. [Free]
† Kate Norlock’s original comment is here.
†† Thanks to Kate Norlock for pointing me to this review.
††† For more on prestige in academia, see this wonderful presentation: “Prestige: The first and final hurdle…“