On the third Thursday of every month, we speak to a recent graduate about their thesis and their studies. This month’s guest is Dr Eleanor Grieve who has a PhD from the University of Glasgow. If you would like to suggest a candidate for an upcoming Thesis Thursday, get in touch.
Title
Conceptualising, measuring and valuing the impact of health technology assessment
Supervisors
Andrew Briggs, Olivia Wu, Hannah Hesselgreaves
Repository link
http://theses.gla.ac.uk/81864
Why do we lack a good understanding of the impact of HTA?
Many reasons, but two weightier issues are that it is methodologically challenging to do and at what level impact should be measured. Firstly, it is difficult to link the benefits of HTA to impact in terms of health gains, with no consensus that this is, indeed, the level on which we should focus. Secondly, impact evaluations address the difference between what happened with and without an intervention. The dominant framework for causality is underpinned by randomised controlled trials (RCTs) and a counterfactual. However, an RCT is not suited to the evaluation of many complex interventions, including HTA, if we are to treat the process as such. That said, there have been several HTA impact frameworks developed and country evaluations undertaken. The objective is not to rank or score HTA bodies, but evaluations should help to improve HTA in any given context, so these are invaluable from which to learn, encourage accountability, and optimise HTA in delivering health outcomes and value for money. Recent reports from INAHTA present the practices of impact assessment among INAHTA members and agencies’ perspectives on factors that facilitate or inhibit the conduct of impact assessment activities.
What theory underlies your proposed framework?
We undertook a realist synthesis – a theory-driven approach – using quantitative data to capture an empirical measure of uptake of a technology following an HTA recommendation, plus qualitative data to understand what has led to this level of implementation. This interlinks to a return-on-investment (ROI) framework using the measure of uptake to estimate the value of HTA. We use net health benefit (NHB) as our measure of impact. Conveying the concepts of potential, realised, and attributable NHB are key to our framework and relates to the level of uptake; potential NHB is everyone who is eligible receiving treatment, realised NHB is the number of patients currently receiving it, and the counterfactual is what we reckon the situation might have been had the HTA not been undertaken. To this end, our focus is on the step beyond decision-making to the realisation of those decisions, which, in turn, we can influence by using theory to unearth how favourable conditions are created for the uptake of HTA recommendations whilst health gains are modelled. Explanation building, using a realist lens, is operationalised through investigating context-mechanism-outcome configurations and drawing on formal theories. Specifically, we drew upon Weiner’s theory of organisational readiness and the PARiHS framework.
Are health outcomes the only thing that matter?
I often refer to this quote:
“The ultimate value of HTA in a health system depends on its contribution to improved health status or increased efficiency rather than to increased knowledge. In this respect, HTA does not differ much from other health technologies and must be subject to the same rigorous standards of evaluation”.
Velasco Garrido et al (2008)
Our understanding is that the impact of HTA on health outcomes is the major gap in the literature. Absolutely, there will be important spillover effects, such as improved administrative systems, governance, and other positive externalities not accounted for. But the theory-driven work was to help with identifying those other intended and unintended effects. Also, the impact of HTA is often quoted in terms of financial benefits generally presented as cost savings, but this is a narrow interpretation of HTA. HTA is not about cost-cutting or cost-containment, but the efficient use of resources and spending the health budget in a more efficient and accountable manner. Impact evaluations should reflect this.
How realistic are the data requirements for your approach?
Our aim was to utilise routine health administrative data – especially as such data, where it exists, are often underutilised – and HTA outputs that are available in the public domain. In order to get to the overall value of investing in HTA at a systems (or country) level, we need to look at what the process is delivering. In other words, we quantify the value stemming from each individual HTA. Two illustrative case studies were undertaken as part of the thesis, of HTAs at this individual level. Both drew upon existing data and analyses in the public domain, which we then interrogated further and adapted to populate the framework. A natural progression of the case studies would be to apply the full framework to a given jurisdiction. However, we recognise that operationalising the framework to assess the ROI of an entire country’s HTA programme is unlikely to be feasible in any context. Practically, we can only undertake illustrative case studies. Rather than utilising data from existing systems, a full application of the framework would more likely require a specific evaluation or audit to be commissioned to purposely collect the necessary data.
What did your case studies reveal about the value of HTA?
On the quantitative side, value for money, with both case studies showing their costs (expressed in terms of their health equivalence) substantially offset by their benefits to different degrees. We make explicit that we present these case studies as illustrative only; as proof-of-method studies rather than for their empirical significance, with key simplifying assumptions underlying the framework clearly stated. We thus acknowledge uncertainty in the empirical findings. The theory-based evaluation indicated that where there was an interaction between collective change efficacy or change commitment and capability, this led to more successful adherence or uptake of HTA decisions. This was more often achieved where there was a greater connectedness between HTA and health systems, showing a concern for the values and interests of the actors with the potential to block or subvert implementation and their interactions with the health system. We suggest two practical ways to do this. First, a refocus on meso HTA (for example, guidelines to manage patient care pathways within a healthcare system) or macro HTA (efficiency, organisation, and strengthening of the healthcare system), rather than the more traditional micro HTA. Second, greater use of early (or development-focused) HTA to deal with context-dependent evidence more scientifically and less ‘colloquially.’
What would be necessary to see your approach widely adopted?
I presented this work to an HTA agency recently and put this question straight back to the experts. I came at this from the perspective of donors, such as the Gates Foundation and the International Decision Support Initiative investing in HTA in lower-income countries, which have generated greater interest in the ROI of HTA. So, I am keen to know too how to broaden this. Where HTA is institutionalised, might this approach be attractive to show the benefit of HTA to those governments? As stated, it would not be practical to evaluate a whole HTA programme, but perhaps the framework could be applied to cluster case studies, for example, in a specific disease area to show the impact of what has been done in related technologies. In turn, this could help inform investment – either where HTA makes the most impact or, conversely, the hard-to-reach areas. Going forward (post-COVID), investment will be heavily scrutinised. Demand to spend better is greater than ever. HTA is all about investment, regardless of the budget, but the fixed and tight budget is ever more apparent. Impact assessment can look at those decisions to help think about prioritisation and disinvestment – absolute gains.
