Ben Goldacre, author of Bad Science, recently published an article online for his Guardian column discussing hospital level variation in Bowel Cancer mortality. In this article he discusses how journalists often see patterns in random variation, to which he responds by producing a funnel plot of standardised mortality ratios for bowel cancer mortalities which demonstrates that the mortality in the vast majority of hospitals is within expected limits. However, this article only highlights a problem with hospital performance ranking, that these rankings only demonstrate an association and no causal effect. This is a problem since, for example, the fact that patient choice is affected by performance rankings means that hospital choice is an endogenous one.
The purpose of ranking hospitals is to identify which ones are not achieving the results expected of them given their patient case-mix. Or, to put it differently, the aim is to identify the treatment effect of each hospital. For a typical medical intervention we might perform a randomised controlled trial to estimate the causal effect of treatment, however for whatever reason, this is not always possible. For hospitals we lack the counter-factual against which we can compare each treated patient. Can we say that a particular patient who died at an ‘underperforming’ unit would have been less likely to die at a ‘better performing’ one?
A key problem is that hospital choice is endogenous. Certain patients are more likely to be treated in different hospitals. A more severe patient may be transferred to a larger unit. Higher income patients may choose to, since they are more able to, travel to better rated or private hospitals. In areas with a wide income disparity we may be left with a greater proportion of low socioeconomic status individuals, consider Glasgow which often appears to be ‘underperforming’. Hospital rankings then can exacerbate this situation by altering patient preferences.
There are a number of possible tools to examine causal inference, and often a subjective interpretation of the problem is required. Since these techniques lie beyond most journalists’ capabilities we shouldn’t expect to see causal inference appearing in the media any time soon. Agencies, such as Dr Foster, whose results are widely published in the media, generally only use a few covariates such as age, sex, deprivation score, and comorbidity index. Individual and hospital level effects are rarely separated. In order to estimate the actual effect a hospital is having, further, more in depth analysis is required. Choice in the health care market is affected by studies of the health care market, and if these studies make mistakes then the market will be distorted and not achieve its optimal outcome.
