Identifying the effect of expenditure on health outcomes: another small comment on Claxton et al

In a previous post I asked whether the study by Claxton et al can or should inform the cost-effectiveness threshold used by NICE. The authors argued that, “it is the expected health effects … of the average displacement within the current NHS … that is relevant to the estimate of the threshold.” Accepting this premise, the authors aimed to estimate the average net effect (in terms of QALYs) that has historically resulted from contraction and expansion of the healthcare budget. I want to explore here briefly whether their empirical estimates can indeed be interpreted as such.

When the healthcare budget contracts we may remove services or technologies with a low cost-effectiveness and when it expands we may implement services or technologies with a high cost-effectiveness. The argument is that we don’t want to reimburse a new technology that has a lower cost-effectiveness than what the healthcare service already achieves in practice when the budget changes, which is the net effect of contraction and expansion. The approach taken by Claxton and colleagues to estimate this effect was to use local healthcare authority level data on healthcare expenditure across different programmes of healthcare (e.g. respiratory healthcare or oncology) and examine how changes in this expenditure affected healthcare outcomes, such as mortality. Since healthcare expenditure is likely to be correlated with unobservable determinants of mortality, the authors adopted an instrumental variables approach.

At this point it is important to note that total healthcare expenditure may vary for two reasons. On the supply side, there may be changes in unit costs or shifts in the overall budget constraint; on the demand side, population health may change affecting the need for healthcare, the identity of the patients, and the types of services demanded as well as reductions in the use of healthcare by current patients. I would argue that it is the supply side changes that we are interested in here. Demand side changes may shift which programmes are utilised, and the resulting productivity of those programmes, since the characteristics of the patients will change.

The estimates from an instrumental variables estimator can be interpreted as the local average treatment effect (LATE) which is the average effect of a change in the variable of interest (in this case healthcare expenditure) resulting from a change in the instrumental variable (IV). The IVs utilised by Claxton et al are socio-economic variables (such as the index of multiple deprivation and the proportion of the population providing unpaid care). These variables are arguably on the demand and supply sides since they both affect population healthcare needs leading to different populations being treated and may affect the healthcare utilisation of current patients.

The empirical estimates of Claxton et al may therefore possibly be interpreted as the effect of both changes due to contractions and expansions of the budget (the effect of interest) and a change in the programmes of care provided, the treatments within them, and their productivity resulting from changes to population health needs and the identity of patients.*

Overall, I think that even if we accept the authors’ arguments about why they are trying to identify this effect, their empirical strategy may possibly not identify it.


*It may be argued that a test of over-identifying restrictions (OID), which tests if the instruments are correlated with the errors, would detect if these instruments were related to health needs. However, note that we are looking within a programme of care, for example, cancer expenditure on cancer mortality so that we are conditioning on having cancer (and being diagnosed with it) in this analysis. Socio-economic variables may be determinants of getting cancer or which type of cancer a patient gets but may not be determinants of (i.e. are independent of) the health outcomes from cancer once we’ve conditioned on having cancer and some other factors determining health outcomes. They may therefore pass the OID test.


Tags: , , , , , , , , , , ,

Do we really need to change the cost-effectiveness threshold?

The cost-effectiveness threshold utilised by health technology assessment agencies, such as NICE in the United Kingdom, below which new medical technologies and interventions are considered cost-effective, is frequently discussed. NICE currently use a threshold of £20,000 to £30,000 per quality adjusted life year (QALY) gained. However, this threshold was arrived at in somewhat of an ad hoc manner, being simply a reflection of past recommendations made by the agency. As a result, much time has been spent trying to identify what the threshold ought to be in order to best ensure the efficiency of the health service.

To much media attention, a study by Claxton et al was published recently, in which the authors attempt to estimate the returns currently being achieved by the NHS in England. The argument goes that the consideration to adopt a new medical technology should take into account the opportunity cost of doing so; any new technology, given a fixed budget, will displace resources used to achieve health benefits elsewhere in the healthcare service. If the new technology is not as cost-effective as the returns currently being achieved for the money, then the overall efficiency of the healthcare service will be reduced. Claxton and colleagues arrived at the figure of approximately £13,000 per QALY and have argued that NICE should adopt this as their threshold.

The opportunity cost is the benefit foregone by not spending a certain amount of money or deploying resources elsewhere. Within the health service we do not want to reimburse a new treatment when we could alternatively use those resources to achieve greater health gains otherwise; hence, the argument that the threshold should reflect the opportunity cost. What then is the opportunity cost in the health service? Assuming a fixed budget, it is the health gains made from increasing the expenditure on the most cost-effective programme of care minus the health losses from reducing the expenditure on the least cost-effective programme of care, when the healthcare budget is contracted and then expanded.

Does the Claxton et al study estimate this opportunity cost? Only if we assume that there is allocative efficiency in the healthcare service, i.e. that the most cost-effective programme funded when the budget is expanded has the same cost-effectiveness as the least cost-effective programme removed when the budget is contracted, and when there is optimal displacement, i.e. that the displaced technologies are the least cost-effective (Eckerman and Pekarsky, 2015). Neither of these conditions are likely to hold in the health service given the nature of the healthcare market, which may suggest that the Claxton et al results are underestimates of the true opportunity cost.

The above discussion assumes that the goal is the maximisation of population health. However, equity considerations play a role in reimbursement decisions such that we might be willing to maintain funding for a less cost-effective service if it preserves some measure of equity. Incorporation of equity concerns into economic evaluation is often not done in practice but methods do exist. In such a case, we may wish to adopt an equity weighted threshold that reflects an equity weighted opportunity cost. Alternatively, we could allow a different threshold for different patient groups, where the difference between the thresholds reflects society’s willingness to pay for benefits accruing to different persons. Either way we may prefer a threshold higher than the Claxton et al  figure to make room for equity considerations.

A final point is that profit-maximising manufacturers strategically price their products at the cost-effectiveness threshold. Under these conditions, even if displacement is optimal, then there will be no net gain to population health from adopting the new products despite them meeting the cost-effectiveness threshold.

What this all may suggest is that, methodological issues aside, the Claxton et al study does not provide us with strong enough evidence to change the cost-effectiveness threshold. Further research is required to understand which services are actually displaced, the cost-effectiveness of services currently utilised, and incorporation of equity considerations in reimbursement decisions.

Update: As an addendum and in response to a comment below, Claxton et al do write that, “Given NICE’s remit, it is the expected health effects … of the average displacement within the current NHS … that is relevant to the estimate of the threshold.” This average effect, they arguably do estimate; nevertheless, I think it is important to note that under allocative inefficiency and suboptimal displacement, setting this as the threshold may possibly lead us to either (i) reimburse technologies that are worse than the best alternative (the opportunity cost), or (ii) reject technologies that are more cost-effective than the least cost-effective technology removed under a budget contraction.


Tags: , , , , , , , ,

#HEJC for 26/02/2015

The next #HEJC discussion will take place Thursday 26th February, at 11pm London time on Twitter. To see what this means for your time zone visit or join the Facebook event. For more information about the Health Economics Journal Club and how to take part, click here.

The paper for discussion is a working paper published by the Canadian Centre for Health Economics (CCHE). The authors are Koffi-Ahoto Kpelitse, Rose Anne Devlin and Sisira Sarma. The title of the paper is:

The effect of income on obesity among Canadian adults

Following the meeting, a transcript of the Twitter discussion can be downloaded here.

Links to the article



Summary of the paper

This is the first paper to examine the causal relationship between income and obesity in the Canadian context. To do so, they examined data from five biennial Canadian Community Health Survey (from 2000/01 to 2009/10), a nationally representative survey collecting information on over 100,000 individuals each survey.

Initially, the paper explored the Grossman model, which suggested increasing income would promote healthy lifestyle investments, and thus lead to a negative relationship between income and obesity. Previous studies that examined this link were discussed, some (eg. Lindahl (2005)) demonstrating a negative relationship; some (eg. Schmeiser (2009)) demonstrating a positive relationship; some (eg. Cawley (2010)) finding no evidence of a causal relationship.

Additionally, education and employment were explored. Again, the Grossman model was used as a basis, predicting i) a negative relationship between education level and obesity with a greater income effect amongst educated people and ii) a negative relationship between employment level and obesity. However, regarding education, prior studies discussed have shown “mixed results”, and regarding employment, the authors were not aware of any study to examine this causal relationship, but suggested the relationship was ambiguous.

Finally, the relationship between gender and obesity were discussed. Numerous studies have shown negative association between income and BMI amongst women, but for men, the relationship is unclear (some showing positive relationship, some negative, and some no significant relationship at all). The importance of the effect of obesity on labour market outcomes (outlining the “large” empirical literature showing obese women more likely to suffer discrimination in the labour market) was outlined.

In this study, the authors found that:

  • From 2000/01 to 2009/10, BMI and obesity rates amongst both men and women have risen.
    • For men, the obesity rate rises from 19.48% for those with income below $10k to 26.09% for those with income over $80k.
    • For women obesity falls from 26.71% for those below $10k to 17.38% for those with income over $80k.
  • For men, a 1% rise in household income leads to 0.027 point decrease in BMI (2SLS estimate); 0.084kg reduction and 0.27% point decrease in probability of being obese (linear IV procedure).
  • For women, a 1% rise in household income leads to 0.113 point decrease in BMI (much higher than for men; this used a 2SLS estimate); 0.300kg reduction; and 0.76% point decrease in probability of being obese (linear IV procedure).
  • For men the effect of income on BMI was only demonstrated at higher BMI distribution, while for women the effect of income on BMI was found throughout with a larger effect at higher BMI.
  • Education had a variable relationship amongst both men and women, not consistent with the theoretical prediction that the effect would be larger amongst educated people.
  • The effect of employment for men was mixed, with a negative effect of income on BMI only in employed men and a negative effect of income on obesity probability only in unemployed men.
  • The effect of employment for women was more consistent with theoretical predictions, showing negative effects of income on both BMI and on the probability of being obese across employment status.
  • Higher BMI and probability of obesity was associated with older age, marriage (much greater effect in women), household size (much greater effect in women) and home ownership.
  • Lower BMI and probability of obesity was associated with being widowed/separated/divorced, being an immigrant and living in urban area (in men).

In summary, this study supports the findings of Lindahl, and stands in contrast to Schmeiser, Cawley and other related studies.

Discussion points

  • Why might there be significant variation in findings between the different studies discussed?
  • Are there ways in which unemployment and neighbourhood income might directly influence BMI?
  • Is the set of control variables used in the authors’ models satisfactory?
  • Is it of concern that policies to increase household income could be regarded a pure, explicit public health policy?
  • Are there relevant studies from other countries?
  • To what extent are these findings generalisable?

Can’t join in with the Twitter discussion? Add your thoughts on the paper in the comments below.


Tags: , , , , , , , , , , , , , ,

Health economics journals and negative findings

Recently, a number of health economics journals (henceforth HEJs) co-signed a statement about the publication of negative findings:

The Editors of the health economics journals named below believe that well-designed, well-executed empirical studies that address interesting and important problems in health economics, utilize appropriate data in a sound and creative manner, and deploy innovative conceptual and methodological approaches compatible with each journal’s distinctive emphasis and scope have potential scientific and publication merit regardless of whether such studies’ empirical findings do or do not reject null hypotheses that may be specified.

There was an outpouring of support for this statement; on Twitter, at least. Big deal. Welcome to the 21st century, health economics. Thanks for agreeing to not actively undermine scientific discourse. Don’t get me wrong, it is of course a good thing that this has been published. Inter-journal agreements are rare and valuable things. But is there really anything to celebrate?

Firstly, the statement has no real substance. The HEJs apparently wish to encourage the submission of negative findings, which is nice, but no real commitments are made. The final sentence reads, “As always, the ultimate responsibility for acceptance or rejection of a submission rests with each journal’s Editors.” So it’s business as usual.

Secondly, one has to wonder whether this is an admission that at least some of the HEJs have until now been refusing to publish negative findings. If they have then this statement is somewhat shameful, if they haven’t then it is just hot air.

Thirdly, is publication bias really a problem in the health economics literature? Generally I think health economists – or those publishing in health economics journals – are less committed to any intervention that they might be evaluating, and less rests on a ‘positive’ result. When it comes to econometric studies or issues specific to our sub-discipline I see plenty of contradictory and non-significant findings being published in the HEJs.

Finally, and most importantly for me, this highlights what I think is a great shame for the health economics literature. We exist mainly at the nexus between medical research and economics research. Medical journals have been at the forefront of publishing in a number of aspects: gold open access; transparency; systematic reporting of methods. Meanwhile, the field of economics is a leading light in green open access with the publication of working papers at RePEc, and journals like American Economic Review are committed to making data available for replication studies. Yet health economics has fallen somewhere between the two and is weak in respect to most of these. It isn’t good enough.

There are exceptions, of course. There are a growing number of working papers series. The likes of CHE and OHE have long been bastions in this regard. And there are some journals – including one of the signatories, Health Economics Review – that are ahead of their associates in some respects.

But in general, the HEJs are still on the wrong side of history. So rather than addressing (and in such a weak way) an issue that has been known about for at least 35 years, the HEJs should be taking bolder steps and pushing for progress in our mouldy old system of academic publishing. Here are a few things that I would have celebrated:

  • A commitment to an open-access-first policy. This could take various forms. For example, the BMJ makes all research articles open access. A policy that I have often thought useful would be for html versions of articles to be open access, possibly after a period of embargo, and for PDFs to remain behind a paywall. Journals could easily monetise this – most already deliver adverts. The journals should commit to providing reasonably priced open access options for authors. In fairness, most already do this, but firm commitments are valuable. Furthermore, the journals should commit to providing generous fee waivers for academics without the means to pay.
  • A commitment to transparency. For me, this is the most pressing issue that needs addressing in academic publishing. It’s a big one to address, but it can be tackled in stages. I’ve written before that decision models should be published. This is a no-brainer, and I remain dumbfounded by the fact that funders don’t insist on this. If you have written a paper based on a decision model I literally have no idea whether or not you are making the results up unless I have access to the model. The fact that reviewers tend not to be able to access the models is outrageous. The HEJs should also make the sorts of commitments to transparent reporting of methodologies that medical journals make. For example, most medical journals (at least in principle) do not publish trials that are not prospectively registered. The HEJs should encourage and facilitate the publication of protocols for empirical studies. And like some of the economics journals they should insist on raw data being made available. This would be progress.
  • Improving peer review. The system of closed pre-publication peer review is broken. It doesn’t work. It can function as part of a wider process of peer review, but as the sole means of review it stinks. There are a number of things the HEJs should do to address this. I am very much in favour of open peer review, which makes journals accountable and can expose any weaknesses in their review processes. The HEJs should also facilitate post-publication peer review native to their own journal’s pages. Only one of the signatories currently provides this.

If you are particularly enamoured of the HEJs’ statement then please share your thoughts in the comments below. My intention here is not to chastise the HEJs themselves, but rather the system in which they operate. I just wish that the HEJs would be more willing to take risks in the name of science, and I hope that this is simply a first baby step towards grander and more concrete commitments across the journals. Until then, I will save my praise.

Leave a comment

Posted by on February 9, 2015 in News


Tags: , , , , , , , , , , , , ,

Sofosbuvir: a fork in the road for NICE?

NICE recently completed their appraisal of the hepatitis C drug sofosbuvir. However, as has been reported in the media, NHS England will not be complying with the guidance within the normal time period.

The cost of a 24 week course of sofosbuvir is almost £70,000. Around 160,000 people are chronically infected with the hepatitis C virus in England, so that adds up to a fair chunk of the NHS budget. Yet the drug does appear to be cost-effective. ICERs differ for different patient groups, but for most scenarios the ICER is below £30,000 per QALY. In the NICE documentation, a number of reasons are listed for NHS England’s decision. But what they ultimately boil down to – it seems – is affordability.

The problem is that NICE doesn’t account for affordability in its guidance. One need only consider that the threshold has remained unchanged for over a decade to see that this is true. How to solve this problem really depends on what we believe the job of NICE should be. Should it be NICE’s job to consider what should and shouldn’t be purchased within the existing health budget? Or, rather, should it be NICE’s job simply to figure out what is ‘worth it’ to society, regardless of affordability? This isn’t the first time that an NHS organisation has appealed against a NICE decision in some way. Surely, it won’t be the last. These instances represent a failure in the system, not least on grounds of accountability for reasonableness. Here I’d like to suggest that NICE has 3 options for dealing with this problem; one easy, one hard and one harder.

The easy option

The simplest option involves the fewest changes to the NICE process. Indeed, it would involve doing pretty much what it does now, only with slightly different (and more transparent) reasoning. In this scenario NICE would explicitly ignore the problem of affordability. Its remit would cease to be the consideration of optimality on a national level and it would ignore the budget constraint. NICE’s remit would become figuring out which health technologies are ‘worth it'; i.e. would the public be willing to purchase a given technology with a given health benefit at a given cost. To some extent, therefore, NICE would become a threshold-setter. The threshold should be based on some definition of a social value of a QALY. This is the easy option for NICE as setting the threshold would be the only additional task to what they currently do. Its threshold might not change all that much, or may be a little higher.

However, even if NICE denies responsibility, clearly someone does need to take account of affordability. Given the events associated with sofosbuvir it seems that this could become the work of NHS England. NHS England could use a threshold based on the budget and current QALY-productivity in the NHS. One might expect NHS England to be in a better position to identify the local evidence necessary to determine appropriate thresholds, which would likely be much lower than NICE’s. It would also be responsible for disinvestment decisions. Given the nationwide remit of NHS England, this would still prevent postcode lotteries. The implication here, of course, is that NICE and NHS England might use different thresholds. Any number of decision rules could be used to determine the result for technologies falling between the two. Maybe this is where considerations for innovation or non-health-related equity concerns belong. It seems probable to me that NICE’s threshold would be higher than NHS England’s, in which case NICE would effectively be advising increases in the health budget. This is something that I quite like the sound of.

The hard option

Personally, I believe that NICE’s failure to justify their threshold(s) is quite a serious failing and undermines the enterprise. The hard option will involve them defining it properly, informed by current levels of QALY-productivity in the NHS. Thus properly adopting a position as a threshold-searcher, and doing the job prescribed to NHS England in the ‘easy option’. NICE guidance would therefore be informed by the current health budget and affordability, and therefore must include guidance on disinvestment. The first stage of this work has already been done. The disinvestment guidance would be the hard part. This argument has already been much discussed, and seems to be what many economists support.

I don’t find this argument entirely compelling, at least not as a solution to the affordability problem. To solve this issue NICE would need to regularly review the current threshold and revise it in light of current productivity and the prevailing health budget. It has no experience of doing this. I believe the task could be more effectively carried out by commissioning organisations (such as NHS England), who are in a better position to oversee the collection of the appropriate data and would have a public responsibility to do so. It might also be politically useful if decisions about affordability were made independently of decisions about value.

The harder option

The harder option is for there to be a paradigm shift in the way NICE – and health economics more generally – operates. It could involve programme budgeting and marginal analysis, or the Birch and Gafni approach. This might just be the best option, but it seems unlikely to happen nationally any time soon.

It’s possible that more cost-effective but unaffordable drugs are in the pipeline. Failure to address the affordability problem soon could seriously undermine NICE.

DOI: 10.6084/m9.figshare.1291123


Tags: , , , , , , , , , , , , , , , , , , , ,

Review: Thrive (Richard Layard, David Clark)

Thrive: The Power of Evidence-Based Psychological Therapies

Hardcover, 384 pages, ISBN: 9781846146053, published 3 July 2014

Amazon / Google Books / Allen Lane

Mental illness reduces national income by about 4%, and yet we only spend about 13% of our health budget and about 5% of our medical research funds on tackling the problem.

As an economist who writes a fair bit on mental health, I regularly trot out statements like this about how costly mental health problems are to society and how the under-provision of services is grossly inefficient. To some the point may now seem obvious and trite. As evidence grows ever more compelling, government policy slowly shifts in response. One success story is the Improving Access to Psychological Therapies (IAPT) initiative, which has greatly improved the availability of evidence-based treatment for some of the most prevalent mental health problems in the UK. Yet in many cases we still await adequate action from the government and decision-makers. Two key players in getting IAPT into government policy were Richard Layard – an economist – and David Clark – a psychologist. In their new book Thrive: The Power of Evidence-Based Psychological Therapies, Layard and Clark demonstrate the need for wider provision of cost-effective mental health care in the UK.

The book starts with a gentle introduction to mental illness; what it is, who suffers, the nature of treatment. This will give any reader a way in, with an engaging set-up for what follows (though with one third of families including someone with a mental illness, most people will find the topic relatable). The opening chapters go on to dig deeper into these questions; do these people get help, how does it affect their lives and what are the societal impacts? These chapters serve as a crash course in mental health and though the style is conversational and easily followed, on reflection you’ll realise that you’ve absorbed a great deal of information about mental health. More importantly, you’ll have a deeper understanding. This isn’t simply because of the number of statistics that have been thrown at you, but because of the personal stories and illustrations that accompany the numbers. This forms the first half of the book – ‘The Problem’ – which encourages the reader to start questioning why more isn’t being done. Economists may at times balk at the broad brush strokes in considering the societal ‘costs’ of mental health problems, but the figures are nevertheless startling.

From there the book continues to build. In the second half – ‘What Can Be Done?’ – the authors go on to explain that actually there’s a ton of effective therapies available. We know what they are and who they work for, but they aren’t available. There’s no doubt that the view of the evidence presented is an optimistic one, but it isn’t designed to mislead; where evidence is lacking, the authors say so. The book seems to be written with the sceptical academic in mind; no sooner can you start to question a claim than you are thrown another baffling statistic to chew on. Various therapies are explored, though the focus is undeniably on depression and anxiety and on cognitive behavioural therapy (CBT). Readers with CBT bugbears may feel alienated by this, but should consider it within the broader scope of the book.

Readers would do well to stop after chapter 14. Things go sharply downhill from this point and could, for some readers, undermine what goes before. This would be a great shame. In all seriousness, chapters 15 and 16 would be better off read at a later date, once the rest of the book has been absorbed, understood and – possibly – acted upon. In the final chapters Layard and Clark make distinctly political proposals about how society should be organised. The happiness agenda takes centre stage. In places, mental illness is presented as simply the opposite of happiness. This is an unfortunate and unnecessary tangent. I have some sympathies with the happiness agenda, but for many I expect these chapters would ruin the book. The less said about them the better.

It is a scandal that so many people with mental health problems do not have access to the cost-effective treatments that exist. Layard and Clark demonstrate convincingly that the issue is of public interest. Thrive has the potential to instill in people the right amounts of sympathy, anger and understanding to bring about change. Many will disagree with their prescriptions, but this should not detract from the central message of the book.

DOI: 10.6084/m9.figshare.1287738

Leave a comment

Posted by on January 13, 2015 in Public Health, Reviews


Tags: , , , , , , , , , , , ,

Heterogeneity and Markov models

The big appeal of Markov models is their relative simplicity, with their focus on what happens with a whole cohort, instead of individual patients. Because of this, they are relatively bad at taking into account patient heterogeneity (true differences in outcomes between patients, which can be explained by for example disease severity, age, biomarkers). In the past, there have been several ways of dealing with patient heterogeneity. Earlier this year, I and my co-authors Dr. Lucas Goossens and Prof.Dr. Maureen Rutten-van Mölken, published a study showing the outcomes of these differences in approach. We show that three of the four methods are useful in different circumstances. The fourth one should not be used anymore.

In practice, heterogeneity is often ignored. An average value of the patient population will then be used for any variables representing patient characteristics in the model. The cost-effectiveness outcomes for this ‘average patient’ are then assumed to represent the entire patient population. In addition to ignoring available evidence, the results are difficult to interpret since the ‘average patient’ does not exist. With non-linearity being the rule rather than the exception in Markov modelling, heterogeneity should be taken into account explicitly in order to obtain a correct cost-effectiveness estimate over a heterogeneous population. This method can therefore be useful only if there is little heterogeneity, or it is expected not to have an influence on the cost-effectiveness outcomes.

An alternative is to define several subgroups of patients by defining several different combinations of patient characteristics, and to calculate the outcomes for each of these. The comparison of subgroups allows for the exploration of the effect that differences between patients have on cost-effectiveness outcomes. In our study, subgroup analyses did lead to insight in the differences between the different types of patients, but not all outcomes were useful for decision makers. After all, policy and reimbursement decisions are commonly made for an entire patient population, not subgroups. If a decision maker wants to use the subgroup analyses for decision regarding specific subgroups, equity concerns are always an issue. Patient heterogeneity in clinical characteristics, such as starting FEV1% in our study, may be acceptable for sub-group specific recommendations. Other input parameters, such as gender, race or in our case age, are not. This part of the existing heterogeneity has to be ignored if you use subgroup analyses.

In some cases, heterogeneity has been handled by simply combining it with parameter uncertainty in a probabilistic sensitivity analysis (PSA). The expected outcome for the Single Loop PSA is correct for the population, but the distribution of the expected outcome (which reflects the uncertainty in which many decision makers are interested) is not correct. The outcomes ignore the fundamental difference between the patient heterogeneity and parameter uncertainty. In our study, it even influenced the shape of the cost-effectiveness plane, leading to an overestimation of uncertainty. In our opinion, this method should never be used any more.

In order to correctly separate parameter uncertainty and heterogeneity, the analysis requires a nested Monte Carlo simulation, by drawing a number of individual patients within each PSA iteration. In this way you can investigate sampling uncertainty, while still accounting for patient heterogeneity. This method accounts sufficiently for heterogeneity, is easily interpretable and can be performed using existing software. In essence, this ‘Double Loop PSA’ uses the existing Expected Value of Partial Perfect Information (EVPPI) methodology with a different goal.

Calculation time may be a burden for this method, compared to the other options. In our study, we have chosen a small sample of 30 randomly drawn patients within each PSA draw, to avoid the rapidly increasing computation time. After testing, we concluded that 30 would be a good middle ground between accuracy and runtime. In our case, the calculation time was 9 hours (one overnight calculation) which is not a huge obstacle, in our opinion. Fortunately, since computational speed increases rapidly, it is likely that using faster, more modern computers would decrease the necessary time.

To conclude, we think that three of the methods discussed can be useful in cost-effectiveness research, each in different circumstances. When little or no heterogeneity is expected, or when it is not expected to influence the cost-effectiveness results, disregarding heterogeneity may be correct. In our case study, heterogeneity did have an impact. Subgroup analyses may inform policy decisions on each subgroup, as long as they are well defined and the characteristics of the cohort that define a subgroup truly represent the patients within that subgroup. Despite the necessary calculation time, the Double Loop PSA is a viable alternative which leads to better results and better policy decisions, when accounting for heterogeneity in a Markov model. Directly combining patient heterogeneity with parameter uncertainty in a PSA can only be used to calculate the point estimate of the expected outcome. It disregards the fundamental differences between heterogeneity and sampling uncertainty and overestimates uncertainty as a result.

Leave a comment

Posted by on January 9, 2015 in Economic Evaluation


Tags: , , , , , , , , , , , , , , , ,


Get every new post delivered to your Inbox.

Join 1,334 other followers

%d bloggers like this: