Skip to content

Why I’m quitting peer review

Cognitive dissonance is part of the human condition. I used to eat cheese. But we can do better if we try. I don’t eat cheese anymore, and I won’t engage in pre-publication peer review. I quit.

On numerous occasions, I’ve written about my dislike of pre-publication peer review. It doesn’t work. And yet, I have promoted platforms that support the system, such as Publons. I even joined the editorial board of PharmacoEconomics – Open, and I quite recently became an Associate Editor for Frontiers in Health Services, journals that rely on pre-publication peer review.

I’ve tried to be a good and fair reviewer. I usually identify myself by name in my reports, to allow the victims an opportunity for retaliation. I’ve reviewed or edited more than 100 papers in my time, and I can count on one hand those that I recommended for rejection. Very few papers are devoid of value and beyond salvation. I’ve never been comfortable issuing recommendations for (or against) publication, but still I’ve done it.

I won’t rehearse the criticisms of peer review, as you can read about them elsewhere. What you read here is at least partly inspired by Adam Mastroianni’s article; it encouraged me to finally take action. But I will add that I think health economics is especially constrained by peer review, adopting neither the working paper culture of economics nor the pre-print culture of health sciences. And it’s high time for change.

Authors are not the problem

Perhaps the most important challenge in this context is that we – the researchers – depend on journals for our livelihoods. Journals make the rules. Thus, we remain complicit. I sympathise with the idea of boycotting journals altogether; I once believed this to be the only way forward, and a painful one at that. But it isn’t a necessary step in dealing with the peer review problem. That’s because authors are not the issue here. Peer reviewers and editors are the problem, or at least it is they who sustain the system.

I will – as necessitated by my profession – continue to submit manuscripts to journals that employ peer review. And that doesn’t make me a hypocrite. When I submit to a journal, it is not because I want them to provide me with peer reviews. I’d much rather they didn’t send it for review at all. But it’s their choice. I just want them to publish it, in the hope that my paper might reach people that it otherwise would not.

In choosing to subject our own papers to a journal’s peer review process, we are doing nothing wrong. It’s self-flagellation, sure, but it isn’t contributing to the skewing of the scientific endeavour (unless we sell out and make changes against our better judgement). What is wrong is to subject others to peer review, and doubly so if the process is blind. They don’t deserve flagellation, and certainly not from a masked perpetrator.

What now?

I have handed in my notice (0 days) to PharmacoEconomics – Open and to Frontiers in Health Services; they shall appear ominously on my CV, with an end date. I will no longer provide or oversee pre-publication peer reviews for either journal, or any other. I should say that I still like both journals. Tim Wrightson is one of the best journal editors in our field. The Frontiers model of interactive peer review is far better than almost every other publisher’s. Nevertheless, I want to hasten the transition of both journals to a model that does not include pre-publication peer review.

There are at least three ways in which we can pull our weight without contributing to pre-publication peer review.

First, we can dedicate more time to post-publication peer review. A growing number of platforms and publishers support it. F1000Research is one of the best publishers in this space, and there are platforms like PubPeer. There are also less-joined-up approaches to post-publication review, such as writing about articles on this here blog.

Second, there are a few types of pre-publication peer review that should continue, and I will try to do these more. One is doing your colleagues or mates a favour and having a look at their paper before they submit it. Another is discussing working papers at conferences or other meetings. In this context, authors are not beholden to reviewers and can much more easily ignore feedback if it’s wrong or unhelpful.

Third – and this is hypothetical for me at the moment – we can serve in editorial roles that do not involve pre-publication peer review. In a world without pre-publication peer review, publishers will still need to make editorial decisions, and that’s fine. In the meantime, platforms such as medRxiv or F1000 maintain basic standards to filter out the drivel.

So that’s it. No more pre-publication peer review for me. My morality has found balance; my mind is at ease.

Announcing a new feature

With all that in mind, we are today launching a new feature on the site. Our goal is to encourage more post-publication peer review in the field of health economics. To achieve this, we’ve launched a simple webpage where you can solicit reviews for papers and find papers to review. Visit our new Peer Review page to get involved.


Photo by Mediocre Studio on Unsplash

By

  • Chris Sampson

    Founder of the Academic Health Economists' Blog. Senior Principal Economist at the Office of Health Economics. ORCID: 0000-0001-9470-2369

We now have a newsletter!

Sign up to receive updates about the blog and the wider health economics world.

5 1 vote
Article Rating
Subscribe
Notify of
guest

7 Comments
Newest
Oldest Most Voted
Inline Feedbacks
View all comments
Ewan Gray
Member
8 months ago

Bravo Chris! I think if we did away with peer review as gate-keeper it might not improve the quality of published studies, nor the fairness of selection. But it would certainly improve the timeliness of publication, and would probably give people a more realistic expectation about the fidelity of the reporting and validity of the conclusions. What is important will still be reviewed again post publication. As it often is in subsequent literature!

It would be better if the field understands that the whole concepts of journals, and the position they occupy in determining the academic ‘worth’ of the study, is about curation. It is not a tournament-style meritocracy in which peer-review and journal editors impartially assign scores.

The scientific process will inevitably be skewed by the (mis) judgement of senior scientists. It would be progress to have a system that is at least honest about this.

Paula
Paula
8 months ago

When does this pre-publication period start? Does it include those invaluable discussion presentations at HESG style meetings (what I’ve always referred to as an oral referee’s report)? We do this for free (and pay the cost of conference attendance)?

Nathaniel Hendrix
8 months ago

Admirable step, Chris. I’ve been blackpilled on pre-publication peer review for a long time now, but haven’t been sure exactly what to do about it. I’ve enjoyed some of the blockchain-based ideas around accumulating all your academic activities at a single place, which could even allow for some people developing a niche as peer reviewers.

Incidentally, I’ve noticed that I’ve been getting a lot more peer review requests lately, so I’m wondering if there’s a quiet quitting effect going on that you’re articulating.

Dr Panik
Member
8 months ago

Hi Chris. The new paper review feature on this site is a great idea and I hope it will be a great success.

You said “In a world without pre-publication peer review, publishers will still need to make editorial decisions, and that’s fine.” How will they do that? What criteria will they use? I doubt that any editor will be willing or able to read every submitted paper to judge which ones should be published. And if there is a larger editorial board to do that, won’t that be similar to peer review, just with a smaller pool of people making the judgments? How would they be chosen to ensure diversity of opinions and expertise?

So although there is a lot wrong with peer review in the context of publication in journals, as you’ve convincingly argued, it would be good to think through all of the issues with alternatives. What do you think? I’m perhaps overly influenced by the problems a few years ago with the journal Medical Hypotheses.

Dr Panik
Member
Reply to  Chris Sampson
8 months ago

Your imagined future sounds good, but what should we do about the current reality?  I mis-stated the task that I think editors would be unable to do without peer reviewers.  I’ve never been an editor, but I don’t doubt that they read every paper. (Though I doubt your implied suggestion that they mainly take decisions irrespective of feedback from peer reviewers!)  I was really wondering if they could read every paper in sufficient detail to be able not only to make a yes/no decision but also to offer detailed feedback on why papers are not acceptable or where papers could be accepted if they made changes. These things seem mainly to be outsourced  to peer reviewers. Without assistance from peer reviewers, I think that editors would not only have to be expert on everything in the scope of their journal, but also undertake it as a full-time job. And in effect, you would have one peer reviewer for each and every paper in that journal instead of a few chosen for their specialised knowledge. Quis custodiet ipsos custodes?

Last edited 8 months ago by Dr Panik
7
0
Join the conversation, add a commentx
()
x