Thesis Thursday: Andrea Gabrio

On the third Thursday of every month, we speak to a recent graduate about their thesis and their studies. This month’s guest is Dr Andrea Gabrio who has a PhD from University College London. If you would like to suggest a candidate for an upcoming Thesis Thursday, get in touch.

Title
Full Bayesian methods to handle missing data in health economic evaluation
Supervisors
Gianluca Baio, Alexina Mason, Rachael Hunter
Repository link
http://discovery.ucl.ac.uk/10072087

What kind of assumptions about missing data are made in trial-based economic evaluations?

In any analysis, assumptions about the missing values are always made, about those values which are not observed. Since the final results may depend on these assumptions, it is important that they are as plausible as possible within the context considered. For example, in trial-based economic evaluations, missing values often occur when data are collected through self-reported patient questionnaires and in many cases it is plausible that patients with unobserved responses are different from the others (e.g. have worse health states). In general, it is very important that a range of plausible scenarios (defined according to the available information) are considered, and that the robustness of our conclusions across them is assessed in sensitivity analysis. Often, however, analysts prefer to ignore this uncertainty and rely on ‘default’ approaches (e.g. remove the missing data from the analysis) which implicitly make unrealistic assumptions and possibly lead to biased results. For a more in-depth overview of current practice, I refer to my published review.

Given that any assumption about the missing values cannot be checked from the data at hand, an ideal approach to handle missing data should combine a well-defined model for the observed data and explicit assumptions about missingness.

What do you mean by ‘full Bayesian’?

The term ‘full Bayesian’ is a technicality and typically indicates that, in the Bayesian analysis, the prior distributions are freely specified by the analyst, rather than being based on the data (e.g. ’empirical Bayesian’). Being ‘fully’ Bayesian has some key advantages for handling missingness compared to other approaches, especially in small samples. First, a flexible choice of the priors may help to stabilise inference and avoid giving too much weight to implausible parameter values. Second, external information about missingness (e.g. expert opinion) can be easily incorporated into the model through the priors. This is essential when performing sensitivity analysis to missingness, as it allows assessment of the robustness of the results to a range of assumptions, with the uncertainty of any unobserved quantity (parameters or missing data) being fully propagated and quantified in the posterior distribution.

How did you use case studies to support the development of your methods?

In my PhD I had access to economic data from two small trials, which were characterised by considerable amounts of missing outcome values and which I used as motivating examples to implement my methods. In particular, individual-level economic data are characterised by a series of complexities that make it difficult to justify the use of more ‘standardised’ methods and which, if not taken into account, may lead to biased results.

Examples of these include the correlation between effectiveness and costs, the skewness in the empirical distributions of both outcomes, the presence of identical values for many individuals (e.g. excess zeros or ones), and, on top of that, missingness. In many cases, the implementation of methods to handle these issues is not straightforward, especially when multiple types of complexities affect the data.

The flexibility of the Bayesian framework allows the specification of a model whose level of complexity can be increased in a relatively easy way to handle all these problems simultaneously, while also providing a natural way to perform probabilistic sensitivity analysis. I refer to my published work to see an example of how Bayesian models can be implemented to handle trial-based economic data.

How does your framework account for longitudinal data?

Since the data collected within a trial have a longitudinal nature (i.e. collected at different times), it is important that any missingness methods for trial-based economic evaluations take into account this feature. I therefore developed a Bayesian parametric model for a bivariate health economic longitudinal response which, together with accounting for the typical complexities of the data (e.g. skewness), can be fitted to all the effectiveness and cost variables in a trial.

Time dependence between the responses is formally taken into account by means of a series of regressions, where each variable can be modelled conditionally on other variables collected at the same or at previous time points. This also offers an efficient way to handle missingness, as the available evidence at each time is included in the model, which may provide valuable information for imputing the missing data and therefore improve the confidence in the final results. In addition, sensitivity analysis to a range of missingness assumptions can be performed using a ‘pattern mixture’ approach. This allows the identification of certain parameters, known as sensitivity parameters, on which priors can be specified to incorporate external information and quantify its impact on the conclusions. A detailed description of the longitudinal model and the missing data analyses explored is also available online.

Are your proposed methods easy to implement?

Most of the methods that I developed in my project were implemented in JAGS, a software specifically designed for the analysis of Bayesian models using Markov Chain Monte Carlo simulation. Like other Bayesian software (e.g. OpenBUGS and STAN), JAGS is freely available and can be interfaced with different statistical programs, such as R, SAS, Stata, etc. Therefore, I believe that, once people are willing to overcome the initial barrier of getting familiar with a new software language, these programs provide extremely powerful tools to implement Bayesian methods. Although in economic evaluations analysts are typically more familiar with frequentist methods (e.g. multiple imputations), it is clear that as the complexity of the analysis increases, the implementation of these methods would require tailor-made routines for the optimisation of non-standard likelihood functions, and a full Bayesian approach is likely to be a preferable option as it naturally allows the propagation of uncertainty to the wider economic model and to perform sensitivity analysis.

Simon McNamara’s journal round-up for 1st October 2018

Every Monday our authors provide a round-up of some of the most recently published peer reviewed articles from the field. We don’t cover everything, or even what’s most important – just a few papers that have interested the author. Visit our Resources page for links to more journals or follow the HealthEconBot. If you’d like to write one of our weekly journal round-ups, get in touch.

A review of NICE appraisals of pharmaceuticals 2000-2016 found variation in establishing comparative clinical effectiveness. Journal of Clinical Epidemiology [PubMed] Published 17th September 2018

The first paper in this week’s round-up is on the topic on single arm studies; specifically, the way in which the comparative effectiveness of medicines granted a marketing authorisation on the basis of single arm studies have been evaluated in NICE appraisals. If you are interested in comparative effectiveness, single arm studies are difficult to deal with. If you don’t have a control arm to refer to, how do you know what the impact of the intervention is? If you don’t know how effective the intervention is, how can you say whether it is cost-effective?

In this paper, the authors conduct a review into the way this problem has been dealt with during NICE appraisals. They do this by searching through the 489 NICE technology appraisals conducted between 2010 and 2016. The search identified 22 relevant appraisals (4% of the total). The most commonly used way of estimating comparative effectiveness (19 of 22 appraisals) was simulation of a control arm using external data – be that from observational study or a randomised trial. Of these,14 of the appraisals featured naïve comparison across studies, with no attempt made to adjust for potential differences between population groups. The three appraisals that didn’t use external data were reliant upon the use of expert opinion, or the assumption that non-responders in the intervention single-arm study could be used as a proxy for those who would receive the comparator intervention.

Interestingly, the authors find little difference between the proportion of medicines reliant on non-RCT data being approved by NICE (83%), compared to those with RCT data (86%), however; the likelihood of receiving an “optimised” (aka subgroup) approval was substantially higher for medicines with solely non-RCT data (41% vs 19%). These findings demonstrate that NICE do accept models based on single-arm studies – even if more than 75% of the comparative effectiveness estimates these models were based on were reliant upon naïve indirect comparisons, or other less robust methods.

The paper concludes by noting that single-arm studies are becoming more common (50% of the appraisals identified were conducted in 2015-2016), and suggesting that HTA and regulatory bodies should work together, to develop guidance on how to evaluate comparative effectiveness based on single-arm studies.

I thought this paper was great, and it made me reflect on a couple of things. Firstly, the fact that NICE completed such a high volume of appraisals (489) between 2010 and 2016 is extremely impressive – well done NICE. Secondly, should the EMA, or EUnetHTA, play a larger role in providing estimates of comparative effectiveness for single arm studies? Whilst different countries may reasonably make different value judgements about different health outcomes, comparative effectiveness is – at least in theory – a matter of fact, rather than values, so can’t we assess it centrally?

A QALY loss is a QALY loss is a QALY loss: a note on independence of loss aversion from health states. The European Journal of Health Economics [PubMed] Published 18th September 2018

If I told you that you would receive £10 in return for doing some work for me, and then I only paid you £5, how annoyed would you be? What about if I told you I would give you £10 but then gave you £15? How delighted would you be? If you are economically rational then these two impacts (annoyance vs being delighted) should be symmetrical; but, if you are a human, your annoyance in the first scenario would likely outweigh the delight you would experience in the second. This is the basic idea behind Kahneman and Tversky’s seminal work on “loss aversion” – we dislike changes we perceive as losses more than we like equivalent changes we perceive as gains. The second paper in this week’s roundup explores loss aversion in the context of health. Application of loss aversion in health is a really interesting idea, because it calls into question the idea that people value all QALYs equally – perhaps QALYs perceived as losses are valued more highly than QALYs perceived as gains.

In the introduction of this paper, the authors note that existing evidence suggests loss aversion is present for duration of life, and for quality of life, but note that nobody has explored whether loss aversion remains constant if the two elements change together – simply put, when it comes to loss aversion is “a QALY loss a QALY loss a QALY loss”? The authors test this idea via a choice experiment fielded in a sample of 111 Dutch students. In this experiment, the loss aversion of each participant was independently elicited for four EQ-5D-5L health states – ranging from perfect health down to a health state utility value of 0.46.

As you might have guessed from the title of the paper, the authors found that, at the aggregate level, loss aversion was not significantly different between the four health states – albeit with some variation at the individual level. For each health state, perceived losses were weighted around two times as highly as perceived gains.

I enjoyed this paper, and it prompted me to think about the consequences of loss-aversion for health economics more generally. Do health related decision makers treat the outcomes associated with a new technology as a reference-point, and so feel loss aversion when considering not funding it? From a normative perspective, should we accept asymmetry in the valuation of health? Is this simply a behavioural quirk that we should over-ride in our analyses, or should we be conforming to it and granting differential weight to outcomes depending upon whether the recipient perceives it as a gain or a loss?

Advanced therapy medicinal products and health technology assessment principles and practices for value-based and sustainable healthcare. The European Journal of Health Economics [PubMed] Published 18th September 2018

The final paper in this week’s roundup is on “Advanced Therapy Medicinal Products” (ATMPs). According to the European Union Regulation 1394/2007, an ATMP is a medicine which is either (1) a gene therapy, (2) a somatic-cell therapy, (3) a tissue-engineered therapy, or (4) a combination of these approaches. I don’t pretend to understand the nuances of how these medicines work, but in simple terms ATMPs aim to replace, or regenerate, human cells, tissues and organs in order to treat ill health. Whilst ATMPs are thought to have great potential in improving health and providing long-term survival gains, they present a number of challenges for Health Technology Assessment (HTA) bodies.

This paper details a meeting of a panel of experts from the UK, Germany, France and Sweden, who were tasked with identifying and discussing these challenges. The experts identified three key challenges; (1) uncertainty of long-term benefit, and subsequently cost-effectiveness, (2) discount rates, and (3) capturing the broader “value” of these therapies – including the incremental value associated with potentially curative therapies. These three challenges stem from the fact that at the point of HTA, ATMPs are likely to have immature data and the uncertain prospect of long-term benefits. The experts suggest a range of solutions to these problems, including the use of outcomes-based reimbursement schemes, initiating a multi-disciplinary forum to consider different approaches to discounting, and further research into elements of “value” not captured by current HTA processes.

Whilst there is undoubtedly merit to some of these suggestions, I couldn’t help but feel a bit uneasy about this paper due to its funder – an ATMP manufacturer. Would the authors have written this paper if they hadn’t been paid to by a company with a vested interest in changing HTA systems to suit their agenda? Whilst I don’t doubt the paper was written independently of the company, and don’t mean to cast aspersions on the authors, this does make me question how industry shapes the areas of discourse in our field – even if it doesn’t shape the specific details of that discourse.

Many of the problems raised in this paper are not unique to ATMPs, they apply equally to all interventions with the uncertain prospect of potential cure or long-term benefit (e.g. for therapies for the treatment of early stage cancer, public health interventions or immunotherapies). Science aside, funder aside, what makes ATMPs any different to these prior interventions?

Credits

Rita Faria’s journal round-up for 24th September 2018

Every Monday our authors provide a round-up of some of the most recently published peer reviewed articles from the field. We don’t cover everything, or even what’s most important – just a few papers that have interested the author. Visit our Resources page for links to more journals or follow the HealthEconBot. If you’d like to write one of our weekly journal round-ups, get in touch.

Methodological issues in assessing the economic value of next-generation sequencing tests: many challenges and not enough solutions. Value in Health [PubMed] Published 8th August 2018

This month’s issue of Value in Health includes a themed section on assessing the value of next-generation sequencing. Next-generation sequencing is sometimes hailed as the holy grail in medicine. The promise is that our individual genome can indicate how at-risk we are for many diseases. The question is whether the information obtained by these tests is worth their costs and potentially harmful consequences on well-being and health-related quality of life. This largely remains unexplored, so I expect seeing more economic evaluations of next-generation sequencing in the future.

This paper has caught my eye given an ongoing project on cascade testing protocols for familial hypercholesterolaemia. Next-generation sequencing can be used to identify the genetic cause of familial hypercholesterolaemia, thereby identifying patients suitable to have their relatives tested for the disease. I read this paper with the hope of finding inspiration for our economic evaluation.

This thought-provoking paper discusses the challenges in conducting economic evaluations of next-generation sequencing, such as complex model structure, inclusion of upstream and downstream costs, identifying comparators, identifying costs and outcomes that are related to the test, measuring costs and outcomes, evidence synthesis, data availability and quality.

I agree with the authors that these are important challenges, and it was useful to see them explained in a systematic way. Another valuable feature of this paper is the summary of applied studies which have encountered these challenges and their approaches to overcome them. It’s encouraging to read about how other studies have dealt with complex decision problems!

I’d argue that the challenges are applicable to economic evaluations of many other interventions. For example, identifying the relevant comparators can be a challenge in the evaluations of treatments: in an evaluation of hepatitis C drugs, we compared 633 treatment sequences in 14 subgroups. I view the challenges as the issues to think about when planning an economic evaluation of any intervention: what the comparators are, the scope of the evaluation, the model conceptualisation, data sources and their statistical analysis. Therefore, I’d recommend this paper as an addition to your library about the conceptualisation of economic evaluations.

Compliance with requirement to report results on the EU Clinical Trials Register: cohort study and web resource. BMJ [PubMed] Published 12th September 2018

You may be puzzled at the choice of the latest Ben Goldacre and colleagues’ paper, as it does not include an economic component. This study investigates compliance with the European Commission’s requirements that all trials on the EU Clinical Trials Register post results to the registry within 12 months of completion. At first sight, the economic implications may not be obvious, but they do exist and are quite important.

Clinical trials are a large investment of resources, not only financial but also in the health of patients who accept to take part in an experiment that may impact their health adversely. Therefore, clinical trials can have a huge sunk cost in both money and health. The payoff only realises if the trial is reported. If the trial is not reported, the benefits from the investment cannot be realised. In sum, an unreported trial is clearly a cost-ineffective use of resources.

The solution is simple: ensure that trial results are reported. This way we can all benefit from the information collected by the trial. The issue is, as Goldacre and colleagues have revealed, compliance is far from perfect.

Remarkably, around half of the 7,274 studies are due to publish results. The worst offenders are non-commercial sponsors, where only 11% of trials had their results reported (compared with 68% of trials by a commercial sponsor).

The authors provide a web tool to look up unreported trials by institution. I looked up my very own University of York. It was reassuring to know that my institution has no trials due to report results. Nonetheless, many others are less compliant.

This is an exciting study on the world of clinical trials. I’d suggest that a possible next step would be to estimate the health lost and costs from failing to report trial results.

Network meta-analysis of diagnostic test accuracy studies identifies and ranks the optimal diagnostic tests and thresholds for health care policy and decision-making. Journal of Clinical Epidemiology [PubMed] Published 13th March 2018

Diagnostic tests are an emerging area of methodological development. This timely paper by Rhiannon Owen and colleagues addresses the important topic of evidence synthesis of diagnostic test accuracy studies.

Diagnostic test studies cannot be meta-analysed with the standard techniques used for treatment effectiveness. This is because there are two quantities of interest (sensitivity and specificity), which are correlated, and vary depending on the test threshold (that is, the value at which we say the test result is positive or negative).

Owen and colleagues propose a new approach to synthesising diagnostic test accuracy studies using network meta-analysis methodology. This innovative method allows for comparing multiple tests, evaluated at various test threshold values.

I cannot comment on the method itself as evidence synthesis is not my area of expertise. My interest comes from my experience in the economic evaluation of diagnostic tests, where we often wish to combine evidence from various studies.

With this in mind, I recommend having a look at the NIHR Complex Reviews Support Unit website for more handy tools and the latest research on methods for evidence synthesis. For example, the CRSU has a web tool for meta-analysis of diagnostic tests and a web tool to conduct network meta-analysis for those of us who are not evidence synthesis experts. Providing web tools is a brilliant way of helping analysts using these methods so, hopefully, we’ll see greater use of evidence synthesis in the future.

Credits