I reviewed a paper this morning and it took me only 45 minutes to read the entire manuscript and write out my comments to the authors. When I was an early career researcher (ECR), I used to spend a minimum of three hours per review, and sometimes I would take days. Occasionally, as an ECR, I read the manuscript twice and returned to my review multiple times, re-reading it and revising it to make sure my critique was “comprehensive” and “serious.”
You might be thinking, “45 minutes isn’t nearly enough time to review a paper! It’s probably a careless or sloppy review.” I reflected on this question before submitting the review, and concluded that this is probably one of the best critiques I have ever penned. Why? Because I wasn’t looking for problems that weren’t there. It was a damn good paper. Sure, it wasn’t perfect, and I pointed out its weaknesses, but I stopped short of transitioning from “what’s wrong with this paper?” to the unfortunately more common “how would I have done this study differently?”.
This simple dichotomy (“what’s wrong with this paper?” vs “how would I have done this study differently?”) is at the very heart of the poverty of peer review. If reviewers confined themselves to the first question only, then peer review would be fair and constructive. But reviewers do not confine themselves to the first question. Instead, they venture into the second. A study can be methodologically and theoretically sound, but still rejected by the reviewer simply because, in spite of its soundness, it could have been done differently. As we all know, there is any number of solutions to a single problem. Reviewers tend to prefer their own solution to that of the authors.
“A study can be methodologically and theoretically sound, but still rejected by the reviewer simply because, in spite of its soundness, it could have been done differently.”
Reflecting on my own evolution as a reviewer, and later as an editor, I can clearly see a gradual shift away from “you should have done it my way” toward “technically, there’s nothing wrong with the way you’ve done it, even if I would have done it differently.” I suspect (although this is purely anecdotal) that ECRs are more likely to look for problems that don’t exist in a manuscript because they feel it is their “job” to find problems. They might feel that they would be letting the editor (and science) down by not taking a blowtorch to the manuscript.
As I have evolved into a mature researcher, I find myself increasingly recommending “minor revisions” in Round 1. Alas, to date, I’ve never recommended “accept” at R1, primarily because I’ve never found a flawless paper at that stage. But I’m much less bothered now by the fact that the authors’ approach to the study is different from what mine would have been.
Why are mature researchers more relaxed about peer review than ECRs? I’m not sure. Maybe it’s because full professors are too busy with administrative duties to spend time looking for problems that don’t exist. Maybe it’s because full professors possess a better understanding of what constitutes good research, or at least “good enough” research, than ECRs. Whatever the reason, I submit (pun intended) that your chances of succeeding in peer review increase when full professors serve as peer reviewers. This is, of course, an empirical question, and one I’d like to see answered.
Professor Andrew R. Timming
This article is published under a Creative Commons 4.0 License.