The Absurdity of Peer Review

There is much that divides academics (e.g., different fields of study, methodological preferences, theoretical tastes, ect.), but on this we are all united: Reviewer 2 is a villain. There is no disputing this inexorable fact. R2 is a painful thorn in the side of every researcher.

A couple years ago, I attended our departmental Halloween party dressed as Reviewer 2. I wanted to have the scariest costume and I believe I succeeded in that endeavor. You can see the costume here.

Obviously, I was being humorous (or perhaps more facetious) in dressing up as Reviewer 2, but the thing about humor is that we find something funny because it contains a grain of truth. Here are a few of the highlights from my costume. Each nicely illustrates the absurdity of peer review.

“The authors failed to cite the most important studies in the field, including: Reviewer 2 (2012), Reviewer 2 (2013), Reviewer 2 (2014a), Reviewer 2 (2014b), Reviewer 2 (2015), Reviewer 2 (2017), Reviewer 2 (2018), and Reviewer 2 (forthcoming). All of these papers must be extensively cited in any revision.”

We’ve all received a review like this. We instantly know exactly who the reviewer is and we know that this person is just trying to artificially inflate his or her h-index.

“The theoretical framework you use is completely wrong. You need to re-write the whole paper. See Reviewer 2 (forthcoming) for the right theory to use.”

Ah, yes, the old you-used-the-wrong-theory bit. Not only is this a subjective assessment, but it is also bad science because it encourages post-hoc tinkering with hypotheses.

“Your paper makes a contribution to the literature, but only an incremental contribution. Therefore, it should be rejected.”

This is a newer phenomenon that has emerged in recent years. It basically means that you’re saying something new, but not revolutionary. But let’s be honest. Most research nowadays is tautology. We’re just saying the same thing over and over, but using different words to make it sound cool.

“Your sample size (N=22,000) is much too small to draw any useful conclusions from the results.”

This one is also increasingly common. Your data have to be supersaturated these days to get something published in a top journal. The sample sizes (for qualitative and quantitative research) are so large that the vast majority of the data is redundant. But you have to play the game, and if R2 wants more data, then you have to provide more data.

“Even though 99% of papers published in the social sciences make completely obviously and self-evident conclusions, I’m going to single out your paper for rejection . . . because it makes completely obvious and self-evident conclusions.”

I think this critique of peer review is probably the strongest. Scholarship in general seems to be moving towards quantity over quality. The result is that we’re churning out papers that say very little, if anything at all, that is new or exciting. If you ask for my opinion, I’d say that science would be much better served if we published 1 book every 5 years, as opposed to 5 articles every 1 year.

Prof. Andrew R. Timming

This article is published under a Creative Commons 4.0 license.

One thought on “The Absurdity of Peer Review

  1. Hi Andrew,
    The issue is not peer review, but the training and expectations (and sometimes naked conflicts of interest) of those doing it (sometimes in bad faith). Bad faith is bad faith, and nothing to do with peer review. As I pointe out to Chris Greening on Twitter the other day, coercive citation is against COPE guides. Indeed, when you boil it down, it’s corrupt practice, and should be stamped out. Needs to be called out at source. Re: Books versus journal articles, depends on cultural norms in particular domains. Books may be fine in social sciences, but they’d be write only media in biosciences or computer science.



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: