Hawking is right, Jeremy Hunt does egregiously cherry pick the evidence

I’m beginning to think Jeremy Hunt doesn’t actually care what the evidence says on the weekend effect. Last week, renowned physicist Stephen Hawking criticized Hunt for ‘cherry picking’ evidence with regard to the ‘weekend effect’: that patients admitted at the weekend are observed to be more likely than their counterparts admitted on a weekday to die. Hunt responded by doubling down on his claims:

Some people have questioned Hawking’s credentials to speak on the topic beyond being a user of the NHS. But it has taken a respected public figure to speak out to elicit a response from the Secretary of State for Health, and that should be welcomed. It remains the case though that a multitude of experts do continue to be ignored. Even the oft-quoted Freemantle paper is partially ignored where it notes of the ‘excess’ weekend deaths, “to assume that [these deaths] are avoidable would be rash and misleading.”

We produced a simple tool to demonstrate how weekend effect studies might estimate an increased risk of mortality associated with weekend admissions even in the case of no difference in care quality. However, the causal model underlying these arguments is not always obvious. So here it is:

weekend

A simple model of the effect of the weekend on patient health outcomes. The dashed line represents unobserved effects

 

So what do we know about the weekend effect?

  1. The weekend effect exists. A multitude of studies have observed that patients admitted at the weekend are more likely to die than those admitted on a weekday. This amounts to having shown that E(Y|W,S) \neq E(Y|W',S). As our causal model demonstrates, being admitted is correlated with health and, importantly, the day of the week. So, this is not the same as saying that risk of adverse clinical outcomes differs by day of the week if you take into account propensity for admission, we can’t say E(Y|W) \neq E(Y|W'). Nor does this evidence imply care quality differs at the weekend, E(Q|W) \neq E(Q|W'). In fact, the evidence only implies differences in care quality if the propensity to be admitted is independent of (unobserved) health status, i.e. Pr(S|U,X) = Pr(S|X) (or if health outcomes are uncorrelated with health status, which is definitely not the case!).
  2. Admissions are different at the weekend. Fewer patients are admitted at the weekend and those that are admitted are on average more severely unwell. Evidence suggests that the better patient severity is controlled for, the smaller the estimated weekend effect. Weekend effect estimates also diminish in models that account for the selection mechanism.
  3. There is some evidence that care quality may be worse at the weekend (at least in the United States). So E(Q|W) \neq E(Q|W'). Although this has not been established in the UK (we’re currently investigating it!)
  4. Staffing levels, particularly specialist to patient ratios, are different at the weekend, E(X|W) \neq E(X|W').
  5. There is little evidence to suggest how staffing levels and care quality are related. While the relationship seems evident prima facie, its extent is not well understood, for example, we might expect a diminishing return to increased staffing levels.
  6. There is a reasonable amount of evidence on the impact of care quality (preventable errors and adverse events) on patient health outcomes.

But what are we actually interested in from a policy perspective? Do we actually care that it is the weekend per se? I would say no, we care that there is potentially a lapse in care quality. So, it’s a two part question: (i) how does care quality (and hence avoidable patient harm) differ at the weekend E(Q|W) - E(Q|W') = ?; and (ii) what effect does this have on patient outcomes E(Y|Q)=?. The first question answers to what extent policy may affect change and the second gives us a way of valuing that change and yet the vast majority of studies in the area address neither. Despite there being a number of publicly funded research projects looking at these questions right now, it’s the studies that are not useful for policy that keep being quoted by those with the power to make change.

Hawking is right, Jeremy Hunt has egregiously cherry picked and misrepresented the evidence, as has been pointed out again and again and again and again and … One begins to wonder if there isn’t some motive other than ensuring long run efficiency and equity in the health service.

Credits

Weekend effect redux

The ‘weekend effect’ has continued to make headlines since we last posted about it. Last week an open letter to beleaguered Secretary of State for Health Jeremy Hunt was published in The Guardian by a number of prominent scientists and clinicians calling for an inquiry into the seven day NHS policy and the evidence behind it. And on Monday a further article was published in The Guardian that reported on the findings of a new study in the British Journal of Psychiatry in which it was reported that there was no increase in the risk of mortality for mental health patients in England. Rashmi Patel, the lead author of the study, was quoted as saying that in relation to increased mortality associated with weekend admission their study had shown ‘no significant difference’ and,

Our study does not support the need to have more doctors on duty at the weekends in psychiatric hospitals. In fact, if this means having to reduce the provision of doctors during the week to provide more doctors at the weekend, this could harm patient care.

But, this is not necessarily true, and it goes to show the difficulties with interpreting and translating evidence into effective policy.

I think part of the problem lies with the mindset of there either is or there isn’t a weekend effect. Perhaps this dichotomy has been ingrained into our psyches by hypothesis testing and p-values. But, it’s a bad way to think about it; care does differ between the weekend and weekdays therefore it is quite plausible that care quality differs as well. I don’t think many people believe in what we’ll call strong weekendism, which might be described as there being no patient who would experience a different overall health outcome if they are admitted at the weekend or on a weekday. However, some people may take the weak weekendism position, which might state that no patient who was admitted on a weekend and who died would have survived had they been admitted on a weekday. However, only the strong weekendism position necessarily supports a conclusion that the 7-day NHS policy is unwarranted. Thus, the aforementioned open letter to Jeremy Hunt seems to take too strong a line. However I think both the strong and weak positions are too strong, the most plausible position in my view is that care quality is worse at the weekend. It’s just a question of how much.

Once the question is reframed about the magnitude of the effect rather than its existence, we can start debating whether a policy to remedy care quality differences is worth it. Contrary to Rashmi Patel’s claims above, their study may or may not support increasing clinical staff provision at the weekend. The evidence we have previously covered on this blog provides preliminary evidence that it is highly unlikely that the 7-day NHS policy will be cost-effective by any standard measure. This should be the focus of the debate, not whether a weekend effect exists or not.

Credits

The economics of a 7-day NHS

The recently delivered Queen’s speech set out the government’s plan for “a 7-day NHS”. This vision is a reaction to alarming statistics that mortality rates are increased by 11% for patients admitted to hospital on a Saturday, and 16% if admitted on a Sunday, compared to patients admitted during the week. In a recent paper, I (along with my co-authors) examine the evidence base being used to support this policy move in more detail, and estimate the economic consequences in terms of the potential costs and benefits. The paper focuses on emergency hospital admissions, as this is the area in which the majority of these deaths occur and has been the focus of much of the policy debate.

The evidence base for seven day services

The highly quoted figure of a 16% increase in the risk of mortality is in fact a relative risk, which we all know too well can be misleading. When interpreting risk statistics the key piece of information is the baseline level of risk; figures which are omitted from the case for seven-day services. The most recent figures from England put the elevated mortality risk experienced by patients admitted to hospital in an emergency during the weekend at 0.3 percentage points. Whilst by no means trivial, it is doubtful that this alternative interpretation of the statistics would have summoned quite the same passion for a reorganisation of the entire English healthcare system.

The classic confusion between correlation and causation is the next mistake made when interpreting the ‘weekend effect’ literature. The association between reduced staffing levels in hospitals at weekends and elevated mortality has been cited as the root of the problem, despite a lack of causal evidence to this effect. In spite of this absence of supportive evidence, making routine services available seven days a week has been declared as the solution to tackling the observed weekend effect. The crucial question then, is what are the likely costs and benefits of such service extensions?

“it’s about saving lives”

As economists we are familiar with the concept of opportunity cost, yet sadly it appears that politicians and policy makers have yet to grasp this key notion. Regardless of whether seven-day services are funded through a redistribution of current NHS budgets or an injection of new cash, this decision implicitly diverts potential resources away from patients admitted during the week. The average daily volume of patients admitted to hospital in an emergency is significantly higher on weekdays than during weekends. This means that staff would be diverted away from working at times of high patient volumes to times when there are fewer patients needing treatment. Yet these patients from whom resources are diverted away are never mentioned in arguments of fairness or equity. If, as the government suggest, staffing levels really are the key to reducing mortality, then the introduction of seven-day services may well narrow the gap between weekday and weekend mortality rates. However, it could easily do so by causing the weekday death rate to rise.

Potential benefits and costs of seven-day services

As healthcare policies such as seven-day services are funded from the same NHS budget as new treatments, they should be subject to the same cost-effectiveness evaluation as technologies seeking NICE approval. This requires rigorous evaluation of hard evidence, something seemingly neglected in favour of headline-hitting policy promises. In the paper we use the available evidence, albeit somewhat rudimentary, on the costs and benefits of introducing seven-day services in this setting to assess whether the policy change would likely pass a NICE assessment. We do so under the most optimistic assumption that this service change has the potential to completely eradicate the weekend effect.

Using methods described in detail in the paper, we estimate that reducing the mortality rate experienced by patients admitted in an emergency at the weekends to that observed during the week would result in an annual reduction of between 4,355 and 5,353 deaths occurring nationally (ceteris paribus, of course). This translates into a potential health gain of 29,727 – 36,539 QALYs per year if all of these deaths could be averted. Using the NICE threshold of £20,000 per QALY, the NHS should spend no more than £595m – £731m to achieve a health gain of this size.

Whilst the potential benefits of extending services appear large, they must be compared with the additional costs of doing so. Although caution was emphasised when producing the figures, the best available estimates of the costs of implementing seven-day services are those published by the NHS Seven Days a Week Forum. They estimate this to be 1.5% to 2% of total hospital income, equivalent to a 5% to 6% increase in the cost of emergency admissions. This translates to an annual cost of between £1.07bn and £1.43bn, exceeding our estimates of the maximum amount that the NHS should spend to eradicate the weekend effect by a factor of 1.5 to 2.4, or between £339m and £831m. To make matters worse, all of these calculations take place under the rather optimistic assumption that benefits to patients admitted at the weekend could be achieved without any detrimental effect on outcomes for those admitted during the week.

The way forward

Although alarming, the statistics on elevated weekend mortality are insufficient by themselves to justify a policy change towards extending normal hours of operation into the weekend. There is as yet no clear evidence: that seven-day working will, in isolation, reduce the weekend death rate; that lower weekend mortality rates can be achieved without increasing weekday death rates; or that such reorganisation is cost-effective.

A move towards a fully operational NHS service seven days a week has the potential to have impacts beyond reducing mortality, but these must be evidenced if the policy is to be supported. Mere suggestions that it may reduce factors such as readmission rates and hospital length of stay are not enough to justify a policy change, just as the verbal reassurance of a drug manufacturer that their product was able to cure cancer would not alone secure them NICE approval. Rigorous evidence and evaluation is needed in the policy sphere if we are truly to get the best use from our limited NHS resources. Evaluations of the implementation of seven-day services in the thirteen early adopters should be performed before national implementation is considered, just as any potential new treatment would be trialled before approval.

Disclaimer: The views and opinions expressed are those of the author and do not necessarily reflect those of the HS&DR programme, NIHR, NHS or the Department of Health.