Cognitive dissonance is part of the human condition. I used to eat cheese. But we can do better if we try. I don’t eat cheese anymore, and I won’t engage in pre-publication peer review. I quit.
On numerous occasions, I’ve written about my dislike of pre-publication peer review. It doesn’t work. And yet, I have promoted platforms that support the system, such as Publons. I even joined the editorial board of PharmacoEconomics – Open, and I quite recently became an Associate Editor for Frontiers in Health Services, journals that rely on pre-publication peer review.
I’ve tried to be a good and fair reviewer. I usually identify myself by name in my reports, to allow the victims an opportunity for retaliation. I’ve reviewed or edited more than 100 papers in my time, and I can count on one hand those that I recommended for rejection. Very few papers are devoid of value and beyond salvation. I’ve never been comfortable issuing recommendations for (or against) publication, but still I’ve done it.
I won’t rehearse the criticisms of peer review, as you can read about them elsewhere. What you read here is at least partly inspired by Adam Mastroianni’s article; it encouraged me to finally take action. But I will add that I think health economics is especially constrained by peer review, adopting neither the working paper culture of economics nor the pre-print culture of health sciences. And it’s high time for change.
Authors are not the problem
Perhaps the most important challenge in this context is that we – the researchers – depend on journals for our livelihoods. Journals make the rules. Thus, we remain complicit. I sympathise with the idea of boycotting journals altogether; I once believed this to be the only way forward, and a painful one at that. But it isn’t a necessary step in dealing with the peer review problem. That’s because authors are not the issue here. Peer reviewers and editors are the problem, or at least it is they who sustain the system.
I will – as necessitated by my profession – continue to submit manuscripts to journals that employ peer review. And that doesn’t make me a hypocrite. When I submit to a journal, it is not because I want them to provide me with peer reviews. I’d much rather they didn’t send it for review at all. But it’s their choice. I just want them to publish it, in the hope that my paper might reach people that it otherwise would not.
In choosing to subject our own papers to a journal’s peer review process, we are doing nothing wrong. It’s self-flagellation, sure, but it isn’t contributing to the skewing of the scientific endeavour (unless we sell out and make changes against our better judgement). What is wrong is to subject others to peer review, and doubly so if the process is blind. They don’t deserve flagellation, and certainly not from a masked perpetrator.
What now?
I have handed in my notice (0 days) to PharmacoEconomics – Open and to Frontiers in Health Services; they shall appear ominously on my CV, with an end date. I will no longer provide or oversee pre-publication peer reviews for either journal, or any other. I should say that I still like both journals. Tim Wrightson is one of the best journal editors in our field. The Frontiers model of interactive peer review is far better than almost every other publisher’s. Nevertheless, I want to hasten the transition of both journals to a model that does not include pre-publication peer review.
There are at least three ways in which we can pull our weight without contributing to pre-publication peer review.
First, we can dedicate more time to post-publication peer review. A growing number of platforms and publishers support it. F1000Research is one of the best publishers in this space, and there are platforms like PubPeer. There are also less-joined-up approaches to post-publication review, such as writing about articles on this here blog.
Second, there are a few types of pre-publication peer review that should continue, and I will try to do these more. One is doing your colleagues or mates a favour and having a look at their paper before they submit it. Another is discussing working papers at conferences or other meetings. In this context, authors are not beholden to reviewers and can much more easily ignore feedback if it’s wrong or unhelpful.
Third – and this is hypothetical for me at the moment – we can serve in editorial roles that do not involve pre-publication peer review. In a world without pre-publication peer review, publishers will still need to make editorial decisions, and that’s fine. In the meantime, platforms such as medRxiv or F1000 maintain basic standards to filter out the drivel.
So that’s it. No more pre-publication peer review for me. My morality has found balance; my mind is at ease.
Announcing a new feature
With all that in mind, we are today launching a new feature on the site. Our goal is to encourage more post-publication peer review in the field of health economics. To achieve this, we’ve launched a simple webpage where you can solicit reviews for papers and find papers to review. Visit our new Peer Review page to get involved.
Photo by Mediocre Studio on Unsplash
Bravo Chris! I think if we did away with peer review as gate-keeper it might not improve the quality of published studies, nor the fairness of selection. But it would certainly improve the timeliness of publication, and would probably give people a more realistic expectation about the fidelity of the reporting and validity of the conclusions. What is important will still be reviewed again post publication. As it often is in subsequent literature!
It would be better if the field understands that the whole concepts of journals, and the position they occupy in determining the academic ‘worth’ of the study, is about curation. It is not a tournament-style meritocracy in which peer-review and journal editors impartially assign scores.
The scientific process will inevitably be skewed by the (mis) judgement of senior scientists. It would be progress to have a system that is at least honest about this.
When does this pre-publication period start? Does it include those invaluable discussion presentations at HESG style meetings (what I’ve always referred to as an oral referee’s report)? We do this for free (and pay the cost of conference attendance)?
Admirable step, Chris. I’ve been blackpilled on pre-publication peer review for a long time now, but haven’t been sure exactly what to do about it. I’ve enjoyed some of the blockchain-based ideas around accumulating all your academic activities at a single place, which could even allow for some people developing a niche as peer reviewers.
Incidentally, I’ve noticed that I’ve been getting a lot more peer review requests lately, so I’m wondering if there’s a quiet quitting effect going on that you’re articulating.
Hi Chris. The new paper review feature on this site is a great idea and I hope it will be a great success.
You said “In a world without pre-publication peer review, publishers will still need to make editorial decisions, and that’s fine.” How will they do that? What criteria will they use? I doubt that any editor will be willing or able to read every submitted paper to judge which ones should be published. And if there is a larger editorial board to do that, won’t that be similar to peer review, just with a smaller pool of people making the judgments? How would they be chosen to ensure diversity of opinions and expertise?
So although there is a lot wrong with peer review in the context of publication in journals, as you’ve convincingly argued, it would be good to think through all of the issues with alternatives. What do you think? I’m perhaps overly influenced by the problems a few years ago with the journal Medical Hypotheses.
Thanks for sharing your thoughts! You pose some excellent questions, for which I won’t provide good answers. But editors really should be reading every paper that is currently subject to peer review, or else editors are nothing more than publishers’ henchmen tasked with rounding up reviewers.
My prior is that editors’ acceptance decisions would not meaningfully differ with and without pre-publication peer review. That is, the situation is already as you describe it, except we badge papers as ‘peer reviewed’ and suppose that means something special. I don’t think it does.
Besides, the future that I’m imagining is one in which journal publication is entirely decoupled from peer review. In that world, earlier versions of an article will be released and reviewed in the open, and then the various mags can fight over the right to publish the definitive version. They can use whatever criteria they like! As they already do.
Your imagined future sounds good, but what should we do about the current reality? I mis-stated the task that I think editors would be unable to do without peer reviewers. I’ve never been an editor, but I don’t doubt that they read every paper. (Though I doubt your implied suggestion that they mainly take decisions irrespective of feedback from peer reviewers!) I was really wondering if they could read every paper in sufficient detail to be able not only to make a yes/no decision but also to offer detailed feedback on why papers are not acceptable or where papers could be accepted if they made changes. These things seem mainly to be outsourced to peer reviewers. Without assistance from peer reviewers, I think that editors would not only have to be expert on everything in the scope of their journal, but also undertake it as a full-time job. And in effect, you would have one peer reviewer for each and every paper in that journal instead of a few chosen for their specialised knowledge. Quis custodiet ipsos custodes?
I’m trying to kill the guards, starting with myself! The gatekeepers have failed in their task, so in the current reality, we must open the gates. Journals cannot continue to be responsible for certifying papers as ‘acceptable’ or for determining what should and should not appear in the public domain. We must forcibly remove their responsibility for peer review, and we – as researchers – must insist on a more inclusive and more iterative approach, which necessitates open post-publication peer review. Once we have prised this duty from the hands of publishers, journals will be free (as they are now) to publish or not. But all that will ride on the editors’ yes/no decisions will be their own reputations, the publishers’ profits, and the extent to which any given paper is marketed widely. Thus, I don’t much care how those decisions are made.