Sam Watson’s journal round-up for 7th August 2017

Every Monday our authors provide a round-up of some of the most recently published peer reviewed articles from the field. We don’t cover everything, or even what’s most important – just a few papers that have interested the author. Visit our Resources page for links to more journals or follow the HealthEconBot. If you’d like to write one of our weekly journal round-ups, get in touch.

Financing transformative health systems towards achievement of the health Sustainable Development Goals: a model for projected resource needs in 67 low-income and middle-income countries. Lancet: Global Health [PubMedPublished 17th July 2017

Achieving universal health coverage is a key aspect of the UN’s sustainable development goals. However, what this means in practice is complicated. People need to be able to access health services free at the point of use, but once those services are accessed there needs to be sufficient labour, capital, skill, and quality to correctly diagnose and treat them. For many health systems worldwide, this will require large investments in infrastructure and staffing, but the potential cost of achieving these goals is unclear. This article sets out to estimate these costs. Clearly, this is a complicated task – health care systems are incredibly complex. From a basic microeconomic standpoint, one might need some understanding of the production function of different health care systems, and the marginal productivity of labour and capital inputs to these systems. There is generally good evidence of what is effective and cost-effective for the treatment of different diseases, and so given the amenable disease burden for a particular country, we could begin to understand what would be required to combat it. This is how this article tackles this question, more or less. They take a bottom-up costing approach to a wide range of interventions, governance requirements, and, where required, other interventions such as water and sanitation. However, there are other mechanisms at play. At national levels, economies of scale and scope play a role. Integration of care programs can reduce the costs, improve the quality, or both, of the individual programs. Similarly, the levels of investment considered are likely to have relevant macroeconomic effects, boosting employment, income, and subsequent socioeconomic indicators. Credit is due to the authors, they do consider financing and health impacts of investment, and their paper is the most comprehensive to date on the topic. However, their projections (~$300 billion annually) are perhaps more uncertain than they let on, a criticism I made of similar papers recently. While I should remind myself not to let the perfect be the enemy of the good, detailed case studies of particular countries may help me to see how the spreadsheet model may actually translate into real-world changes.

Precommitment, cash transfers, and timely arrival for birth: evidence from a randomized controlled trial in Nairobi Kenya. American Economic Review [RePEcPublished May 2017

A great proportion of the gains in life expectancy in recent years has been through the reduction of childhood mortality. The early years of life are some of the most precarious. A newborn child, if she survives past five years of age, will not face the same risk of dying until late adulthood. Many of the same risk factors that contribute to childhood mortality also contribute to maternal death rates and many low-income countries still face unacceptably high rates of dying for both mother and child. One way of tackling this is to ensure mothers have access to adequate antenatal and postnatal care. In Kenya, for example, the government legislated to provide free delivery services in government health facilities in 2013. However, Kenya still has some of the highest death rates for mother and child in the world. It is speculated that one reason for this is the delay in receiving services in the case of complications with a pregnancy. A potential cause of this delay in Nairobi is a lack of adequate planning from women who face a large number of heterogeneous treatment options for birth. This study presents an RCT in which pregnant women were offered a “precommitment transfer package”, which consisted of a cash transfer of 1000 KSh (~£7) during pregnancy and a further 1000 KSh if women stuck to a delivery plan they had earlier committed to. The transfer was found to increase the proportion of women arriving early to delivery facilities. The study was a fairly small pilot study and the results somewhat uncertain, but the intervention appears promising. Cost-effectiveness comparisons are warranted with other interventions aiming to achieve the same ends.

Bans on electronic cigarette sales to minors and smoking among high school students. Journal of Health Economics [PubMedPublished July 2017

E-cigarettes have provoked quite a debate among public health researchers and campaigners as we’ve previously discussed. E-cigarettes are a substitute for tobacco smoking and are likely to be significantly less harmful. They may have contributed to large declines in the use of tobacco in the UK in the last few years. However, some have taken a “think of the children!” position. While e-cigarette use per se among adolescents may not be a significant public health issue, it could lead to increased use of tobacco. Others have countered that those young people using e-cigarettes would have been those that used tobacco anyway, so banning e-cigarettes among minors may lead them to go back to the tobacco. This paper takes data from repeated surveys of high school students in the US to estimate the effects of banning the sale of e-cigarettes to minors on the prevalence of tobacco smoking. Interestingly, bans appear to reduce tobacco smoking prevalence; the results appear fairly robust and the modelling is sensible. This conflicts with other recent similar studies. The authors argue that this shows that e-cigarettes and tobacco smoking are complements, so reducing one reduces the other. But I am not sure this explains the decline since no increase in youth smoking was observed as e-cigarettes became more popular. Certainly, such a ban would not have reduced smoking prevalence years ago. At the very least e-cigarettes have clearly had a significant effect on attitudes towards smoking. Perhaps smoking was on the decline anyway – but the authors estimate a model with state-specific time trends, and no declines were seen in control states. Whatever our prior beliefs about the efficacy of regulating or banning e-cigarettes, the evidence is complex, reflecting the complex behaviour of people towards drugs, alcohol, and tobacco.

Credits

What’s going on at the New England Journal of Medicine?

Editorial policies between the top medical journals differ. Some take a ‘crusading’ view and campaign on contemporary health issues. The BMJ falls into this camp, although this has sometimes led them to take political positions that might be contrary to the evidence. Nevertheless, the editorial agenda of the BMJ is clear, readers know what they are backing. The NEJM on the other hand seems to have adopted a more opaque position.

On the face of it the NEJM seems to support a position of ‘if a randomised controlled trial (RCT) has been conducted and it’s published then that’s the last word on the matter’. Some recent examples illustrate this. Ben Goldacre and colleagues in the COMPARE project received a dismissal of their letters penned to the NEJM that expressed concerns over trials that had not reported on the primary outcomes specified in their protocols or reported different outcomes. The New York Times reports on potential flaws or even misconduct in a mega trial of Xarelto, an anticlotting drug, for which the manufacturers are currently being sued. The NEJM, which published the trial, dismissed the relevance of the claims and defended the trial. And, in a recent, controversial editorial, the NEJM appears to endorse the view that researchers who re-analyse trial data from other studies are ‘research parasites’.

This view is not unique to the NEJM. It reflects a broader view that RCTs are definitive and research in top impact factor journals more so. But, scientists are fallible, and RCTs can be flawed and present biased results. For example, in a study of the top four medical journals 95% of RCTs had some missing data, with a median percentage of 9% dropout, and in many cases adequate missing data methods were not used. Publication should not be the final stage of a piece of research but part of an ongoing process.

Part of the problem may lie with the false dichotomy imposed by hypothesis testing and statistical significance. A treatment either works or does not work or it is safe or it is not safe. But, for the most part, these tests are based solely on asking whether the data are compatible with the hypothesis or whether it’s unlikely. All the other forms of uncertainty are not taken into account such as missing data, a lack of adequate allocation concealment, or a lack of double blinding. The researchers could have chosen any number of different tests or comparisons and the choice could be contingent on the data, potentially rendering the p-value meaningless.

Results from RCTs are used to make important clinical and policy decisions. Scrutiny and debate are essential to ensure that the best decisions are made. This includes allowing for an appropriate representation of the uncertainty surrounding a decision. The trust endowed by a high impact factor should bring a responsibility to ensure that well founded critical or dissenting views on published research are appropriately represented. RCTs should be subject to as much scrutiny as any other form of research. Vioxx should serve as an important reminded of this.

Health economics journals and negative findings

Recently, a number of health economics journals (henceforth HEJs) co-signed a statement about the publication of negative findings:

The Editors of the health economics journals named below believe that well-designed, well-executed empirical studies that address interesting and important problems in health economics, utilize appropriate data in a sound and creative manner, and deploy innovative conceptual and methodological approaches compatible with each journal’s distinctive emphasis and scope have potential scientific and publication merit regardless of whether such studies’ empirical findings do or do not reject null hypotheses that may be specified.

There was an outpouring of support for this statement; on Twitter, at least. Big deal. Welcome to the 21st century, health economics. Thanks for agreeing to not actively undermine scientific discourse. Don’t get me wrong, it is of course a good thing that this has been published. Inter-journal agreements are rare and valuable things. But is there really anything to celebrate?

Firstly, the statement has no real substance. The HEJs apparently wish to encourage the submission of negative findings, which is nice, but no real commitments are made. The final sentence reads, “As always, the ultimate responsibility for acceptance or rejection of a submission rests with each journal’s Editors.” So it’s business as usual.

Secondly, one has to wonder whether this is an admission that at least some of the HEJs have until now been refusing to publish negative findings. If they have then this statement is somewhat shameful, if they haven’t then it is just hot air.

Thirdly, is publication bias really a problem in the health economics literature? Generally I think health economists – or those publishing in health economics journals – are less committed to any intervention that they might be evaluating, and less rests on a ‘positive’ result. When it comes to econometric studies or issues specific to our sub-discipline I see plenty of contradictory and non-significant findings being published in the HEJs.

Finally, and most importantly for me, this highlights what I think is a great shame for the health economics literature. We exist mainly at the nexus between medical research and economics research. Medical journals have been at the forefront of publishing in a number of aspects: gold open access; transparency; systematic reporting of methods. Meanwhile, the field of economics is a leading light in green open access with the publication of working papers at RePEc, and journals like American Economic Review are committed to making data available for replication studies. Yet health economics has fallen somewhere between the two and is weak in respect to most of these. It isn’t good enough.

There are exceptions, of course. There are a growing number of working papers series. The likes of CHE and OHE have long been bastions in this regard. And there are some journals – including one of the signatories, Health Economics Review – that are ahead of their associates in some respects.

But in general, the HEJs are still on the wrong side of history. So rather than addressing (and in such a weak way) an issue that has been known about for at least 35 years, the HEJs should be taking bolder steps and pushing for progress in our mouldy old system of academic publishing. Here are a few things that I would have celebrated:

  • A commitment to an open-access-first policy. This could take various forms. For example, the BMJ makes all research articles open access. A policy that I have often thought useful would be for html versions of articles to be open access, possibly after a period of embargo, and for PDFs to remain behind a paywall. Journals could easily monetise this – most already deliver adverts. The journals should commit to providing reasonably priced open access options for authors. In fairness, most already do this, but firm commitments are valuable. Furthermore, the journals should commit to providing generous fee waivers for academics without the means to pay.
  • A commitment to transparency. For me, this is the most pressing issue that needs addressing in academic publishing. It’s a big one to address, but it can be tackled in stages. I’ve written before that decision models should be published. This is a no-brainer, and I remain dumbfounded by the fact that funders don’t insist on this. If you have written a paper based on a decision model I literally have no idea whether or not you are making the results up unless I have access to the model. The fact that reviewers tend not to be able to access the models is outrageous. The HEJs should also make the sorts of commitments to transparent reporting of methodologies that medical journals make. For example, most medical journals (at least in principle) do not publish trials that are not prospectively registered. The HEJs should encourage and facilitate the publication of protocols for empirical studies. And like some of the economics journals they should insist on raw data being made available. This would be progress.
  • Improving peer review. The system of closed pre-publication peer review is broken. It doesn’t work. It can function as part of a wider process of peer review, but as the sole means of review it stinks. There are a number of things the HEJs should do to address this. I am very much in favour of open peer review, which makes journals accountable and can expose any weaknesses in their review processes. The HEJs should also facilitate post-publication peer review native to their own journal’s pages. Only one of the signatories currently provides this.

If you are particularly enamoured of the HEJs’ statement then please share your thoughts in the comments below. My intention here is not to chastise the HEJs themselves, but rather the system in which they operate. I just wish that the HEJs would be more willing to take risks in the name of science, and I hope that this is simply a first baby step towards grander and more concrete commitments across the journals. Until then, I will save my praise.