Suppose you wanted to demonstrate that something caused harm. There’d be at least three things you’d need: 1) evidence of the cause, some change or difference that might lead to the harm, 2) evidence of the harm, describing its nature and magnitude, and 3) evidence of a statistical association between the cause and the harm. A recently published paper in The Lancet makes the bold claim that NICE recommendations cause harm. However, it does so without any of the three things I’ve mentioned here.
I really wanted to let this paper slide. But then, last week, Anita Charlesworth (who is fantastic) cited the research in all seriousness during her plenary at HESG in Bristol. So here we are.
This blog post is built on the simple premise outlined above – that the authors don’t have the evidence they need to even consider making such a claim about the impact of NICE decisions. I’ll tackle each of the three necessary pieces of evidence in turn. But first, a brief summary of the analysis.
Naci et al (2025)
Let’s be clear: this paper does not proceed with the intention of testing a novel hypothesis or shining a light on some previously obscured truth. Rather, it represents the continuation of a campaign to reduce NICE’s cost-effectiveness threshold. The Discussion section makes this plain in comic style, like responding to the classic “what’s your greatest weakness” job interview question with “I’m too smart and I work too hard and I don’t have time to tell you how true these things are”.
The analysis itself is quite neat and straightforward. There are, in short, two steps.
First, the researchers looked at the cost-per-QALY (ICER) that NICE has approved for new medicines in recent years. These tend to fall within NICE’s stated threshold range of £20-30,000 per QALY. The authors then compare these ICERs to a figure of £15,000 per QALY as an estimate of the NHS’s productivity. Thus, any drug approved above £15,000 per QALY is assumed to cause harm in its use, and the authors estimate this for the average patient taking each newly approved drug.
Second, the authors scale this up to the population health impact by multiplying the per-patient effects (usually harms) by real world evidence on the total number of patients being prescribed the medicine.
That’s how we end up with a headline estimate that NICE-approved drugs generated 3.75 million QALYs, but that alternative use of the funds would have generated 5 million QALYs, such that NICE approvals generated a loss of 1.25 million QALYs.
The cause
The source of harm proposed by the authors is the NICE approvals, but these are not directly harmful in themselves. The authors implicitly assume a mechanism whereby:
- NICE approves a medicine
- NHS commissioners fund its provision
- NHS clinicians prescribe it
- NHS commissioners reduce other spending
In this study, the researchers only observe 1 and 3.
The authors observe NICE’s approvals, not in time or space, but simply through the assertion that we live in a world in which NICE has approved these medicines. The researchers use evidence on prescriptions, which is this study’s most innovative contribution.
The most important part of the mechanism – that which actually has the potential to cause harm – is step 4. For this, we have no evidence. In terms of expenditure, we do not know what happens when NICE approves a medicine. Thus, the entire study relies on the authors’ assumption that the NHS pays the price considered by NICE for each medicine (certainly not true), and that commissioners immediately reduce spending by this amount (also not likely to be true). Regardless of the extent to which you believe it, we can surely agree that this is not an evidence-based assertion.
In short, the researchers claim that NICE approvals reduce spending on things other than the approved medicine, without any evidence that this is true or any attempt to observe this reduction in spending.
The harm
The harm that NICE is supposedly inflicting is estimated in terms of QALYs, or rather QALYs foregone. I have no qualms about considering an opportunity cost to be a harm.
This study does not use any real-world evidence on QALYs, foregone or otherwise. We don’t know about population QALYs before the NICE approval, or after the NICE approval, or the means by which any QALYs were generated. There is no counterfactual.
All we have is the QALY estimates used in the relevant NICE approvals and the assumption that every £15,000 that is displaced results in a displaced QALY.
In short, there is no evidence of any harm whatsoever.
Statistical association
We have very weak evidence for the existence of the cause, and no evidence of the harm. But let’s pretend, for a moment, that we have strong evidence for both. The other key requisite would be to demonstrate a causal link between the two, otherwise we might be looking at spurious correlations.
What do the researchers give us? The answer is absolutely nothing: not even a hint of an attempt to demonstrate causality. Even if we knew exactly how much was being spent on each new medicine and the QALY gains associated with them, and even if we were 100% certain that NICE recommendations led to disinvestment, we would have no idea about the QALY impact of such disinvestments.
But what about that £15,000 figure? Arguably, that is even less evidence-based than NICE’s £20-30,000 range, which was at least identified on the basis of historic NICE decision-making. £15,000 is approximate to some earlier estimates of the ‘marginal cost per QALY‘ in the NHS, but these estimates are derived from historic regional differences in total expenditure and not budget allocation decisions following the approval of new medicines. There’s no shortage of concerns about the suitability of these estimates for policymaking, but the key point here is that they are not derived from the same context in which the authors of this study are claiming causality.
What now?
Studies like this can be incredibly influential. They provide a convenient and nominally evidence-based excuse for the government to reduce certain types of expenditure. As such, they warrant intense scrutiny.
There’s a bunch of other stuff that we’d want to see beyond the three things I’ve considered in this blog post, including clear articulation of a mechanism for causality, identification of a temporal sequence, and the ruling out of alternative explanations. I’ll leave a full critique of the paper to somebody else.
Of course, I cannot disprove the researchers’ claim. For now, we have no evidence one way or the other. But I would encourage you to consider your instincts, and bring your common sense where the evidence is lacking. Do you think NICE’s health technology assessment work causes harm? My prior is that NICE, NHS commissioners, and NHS clinicians all work hard to make decisions that achieve the best outcomes for patients, and will tend to succeed. This research insists that they are all consistently failing.
None of this is to say that research of this kind isn’t worthwhile. As this blog post describes, there’s a lot that we need to understand in order to assess the health impact of expenditure decisions, and we need to start somewhere. But this paper must be viewed for what it is, a weakly specified study by researchers who seem unable or at least unwilling to consider its shortcomings; shortcomings that will prove convenient to ignore for policymakers.
One day, we may know the truth about these things, and be able to make the sorts of claims that the authors of this study wish to make. But if you prefer your assertions to be evidence-based, you may safely dismiss this paper.
Well done on the absence of evidence and the focus on thresholds (often ignored when US writers talk about QALYs.) But excuse my ignorance how could there be a counter factual? Is that a real argument or is it like if I were Ariana Grande I would have a great voice?
A counterfactual in this case could be commissioners not funding (or having not yet funded) the new medicine. That might be based on data from the time before the recommendation was issued, or perhaps from a region where commissioners have found a way to delay adoption of the technology.