Chris and I differ in several ways. For starters, I still eat cheese. And, to misquote the late Charlton Heston, you can pry it from my cold, dead hands. I will also continue to peer review pre-publication manuscripts, although my defence of doing so won’t be as vehement as for dairy product consumption.
To begin by setting out my reviewing experience, I’ve done probably around 50 reviews for 15-20 different journals. I don’t have much experience of sitting on an editorial board, having joined The Patient’s recently enough not to regret it yet. My approach is to be fair and constructive, and my goal is to help the authors improve their manuscript, even in the relatively rare cases where I recommend rejection. I don’t identify myself to authors, but I’m unconcerned if people can work out it’s me. (This is usually pretty easy – my reviews are the ones urging you to cite Webb et al.)
Chris successfully defends himself against hypocrisy for still sending manuscripts to journals with pre-publication peer review while not reviewing himself. Publishing in those journals is a career necessity, and in his view it’s the journal’s decision to use peer review, while he himself would rather do without it. My primary reason for continuing to review is that I would be a hypocrite for not doing so: I want my manuscripts to be peer-reviewed, so it is only fair that I provide reviews for my colleagues.
Dr Strange Comment, or How I Learned to Stop Worrying and Love Reviewer 2
I cannot think of a publication of mine that has not been overall improved by undergoing pre-publication peer review. Even the remark that one of my manuscripts was “awfully dull” was helpful. It made me realise that I hadn’t brought out the interesting aspects of the work, and spurred me to revise the manuscript so that the published version was (hopefully!) less dull. Such feedback could be provided post-publication, of course, but this way I don’t have to worry about putting howlers in the public domain, and readers are spared from slogging through the dull version of the paper.
Another reason for preferring the private and anonymous feedback that pre-publication review affords is that I have greater trust that the reviewers are being honest with their opinions. Anonymity, and the knowledge that only a few people will ever read the comments, is beneficial to those who otherwise might be afraid of putting their heads over the parapet to criticise others’ work publically. This is especially important when junior researchers are often providing critiques of influential professors’ work.
Being a reviewer also has many benefits. Especially for junior researchers, closely reading and critically appraising colleagues’ work can help improve their own practice. Another advantage of reviewing is that, like everyone, I have an ever-expanding to-do list. The commitment to providing a review is extremely helpful in keeping abreast of the current literature.
Given the virtues of peer reviewing I’ve listed above, I don’t believe, as some argue, that pre-publication peer review is fundamentally broken. This is not to say that it is perfect. I have my fair share of horror stories – waiting eight months for a desk rejection, reviewers who clearly haven’t read the paper, etc. Yet I don’t believe this means we should scrap the whole system. In the rest of this post, I talk about why I don’t think we should move to a completely new way of publishing work, as well as some ways we can improve the system we have. (A disclaimer: these are my personal opinions, based on my own experiences, which will inevitably be anecdotal. Your anecdotes may well be different.)
Going postal
Chris argues for replacing pre-publication peer review with post-publication review. The system would have advantages, which Chris clearly articulates. However, I am somewhat sceptical. A key problem with pre-publication review is the length of time the process takes, and I can’t believe that a post-publication system would be any faster at arriving at a definitive version of a paper. If anything, it would be even slower without the impetus to get a published manuscript out. It’s true that a version of a paper would be available sooner. Yet it’s not clear to me that having non-definitive versions of every manuscript floating around would be beneficial. There are, of course, times when disseminating results as soon as possible is crucial, but avenues for doing so, such as working paper series, are already available without dispensing with pre-publication peer review.
Mixed signals
Regardless of the specifics of the proposal, there are two factors which I believe it is important for people trying to create a utopian alternative to the current pre-publication peer review system. First, pre-publication peer review is often criticised for being ‘gatekeeping’. Yet it is important to acknowledge that journals provide useful signals of quality, subject matter, and novelty. The signals are imperfect, and there are certainly issues to address, but given the volume of research produced globally, they are essential and inevitable. In any new system, the research community will find a way of creating new signals to filter out the few articles which a given individual needs to read from the rest.
Illegitimi non carborundum
Second, I believe that any reform of the research publication system must take into account the fact that there are actors involved whose interests are not aligned with those of the research community. There are companies that extract large amounts of rentier income from researchers and universities. I won’t name individual companies, but will rather use the umbrella term ‘bastards’. Given the amount of money at stake, I can’t believe that the bastards wouldn’t find a way of extracting as much cash as possible from any new system. We have already seen this happen with open-access reforms. Rather than opening up research to the public and reducing rip-off subscriptions, it has turned into an expensive paywall where researchers are charged thousands, if not tens of thousands, for the privilege of disseminating their work. I would prefer the bastards didn’t exist, and would welcome any proposals which I thought would get rid of them. But these proposals will need to be good enough to outsmart well-resourced bastards with a lot to lose.
Always look on the bright side of life
To counteract the rather pessimistic views above, I want to conclude with some ways in which we, as a research community, can help improve a far-from-perfect system. First, we should try to find a publication system which works for health economics. We are our own discipline, and what works for physics, or medicine, or ‘economics economics’ won’t necessarily work for us. We should not be afraid of divergence, nor ape other fields for the sake of it.
As a community, we as health economists should discuss more what we would like the peer-reviewing process to look like, and what we believe makes a good review. We should also be better at providing training and guidance for new (and not-so-new) peer reviewers. I have my own opinions about what makes a good review, and how to be a good reviewer. However, I’m also aware that I have many colleagues, far cleverer than I, who probably have much better ideas that I could benefit from. A way of discussing ideas about peer reviewing in health economics would be hugely beneficial. Perhaps HESG could be used as a forum for initial discussion, at least in the UK.
The peer-reviewers’ flag is deepest red
However, the most fundamental problems with the pre-publication peer review system, in health economics and elsewhere, are down to a lack of resources. If editors and reviewers are expected to work for free, it should be no surprise that the work isn’t always of the highest standard. The saying goes that if you pay peanuts, you get monkeys, but most journals don’t even pay peanuts. Reviewing is also expected to be done on top of a full-time job (which in practice is usually considerably more than a full-time job). As reviewing and editing are expected parts of academics’ jobs, they ought to be an explicit part of our contracts, with specific time allocated to them. Perhaps this is somewhat naïve given the funding constraints our institutions are under. But if you don’t ask, you don’t get, and I believe this is something trade unions should be campaigning for.
At the end of the day
Finally, I believe we should be open to new initiatives and ideas to help improve the health economics publication process. Despite largely arguing in favour of the status quo, I hope that the blog’s new peer review page is a success and provides a useful alternative and much-needed competition to the conventional publication process. In the meantime, if you’re continuing to submit via the standard pre-publication review route, remember: you never know who your Reviewer 2 will turn out to be, so make sure you cite Webb et al.
The author would like to thank one anonymous reviewer (Chris) for helpful comments.
