Decision models can be very useful. They can also be very bad. ISPOR and SMDM recently got together to form the Modeling Good Research Practices Task Force and have produced a set of reports. One of the reports, by David Eddy and colleagues, considers Model Transparency and Validation. The authors discuss the importance of transparency and validation in ensuring model credibility. They provide an excellent set of best practices that all modellers should follow. Go and read it.
The problem
The apparent effectiveness of a health care intervention can be distorted in many ways. Ben Goldacre‘s new book does a pretty good job of covering these. When it comes to cost-effectiveness the opportunities possibilities for distortion increase exponentially. A recent paper investigated FDA actions against health economic promotions. Many companies seek to promote products based on their economic value of “saving money” or “lowering costs”. The most common type of economic violation was an implied claim of cost savings due to work productivity or functioning. One can imagine how an unjustified figure of this nature could go unrecognised in a model.
There will always be intentionally misleading models, with which regulators will have to deal. However, most researchers’ models will have validity issues in spite of good intentions. If Eddy et al’s best practices were adhered to by good researchers then bad models could be easily identified and amended. The authors argue that, to ensure transparency, each model should be accompanied by both technical and non-technical documentation available to interested readers. However, a feasible process of ensuring transparency in this way does not currently exist.
The solution
The solution is to establish a model registry and database. It is unrealistic to expect journal publishers to accept extensive appendices, to which additions would need to be made as a model is developed.
Setting up a registry and online database would be very simple. Inclusion in the registry should require a minimum submission to the database: Eddy et al’s proposed non-technical documentation. Further information and files could be optional. Each model, whether single-application or multi-application, would be assigned a registration number and a corresponding database entry and webpage. The webpage could include all of the technical and non-technical documentation required by Eddy et al’s best practices. It could also include relevant citations, manuals and code. The site should provide a means for feedback and discussion. As models are updated and improved, or used in other applications, details can be added about subsequent versions. I’m not convinced of the argument presented by Eddy et al regarding intellectual property. This is meant to be science; one wouldn’t expect a health researcher to keep their trial design secret. Regardless, a registry would further protect intellectual property rights. Copyright is an automatic right in most countries, and plagiarism or use without appropriate attribution could easily be recognised and acted upon in a transparent system. The registry would provide a means of ensuring appropriate attribution of intellectual property.
There will be costs associated with maintaining such a registry and database (web hosting, doi fees), but most would be achieved by the contribution of researchers’ time. Academics are well accustomed to pro bono work and this shouldn’t represent a serious barrier. The primary challenge would be in encouraging researchers to submit their models to the registry. Model registration could not be a pre-requisite for publication in the same way as trial registration, so researchers would have to see other incentives. I would see the primary benefit as increased citations. If people are allowed to use existing models this will lead to further citation of the original, which could be facilitated by the provision of a Digital Object Identifier and would promote model validation.
An open and transparent registry would enable the research community to investigate the validity of models, which may be used to inform policy decisions. A registry would also help prevent the duplication of efforts. So, I call on relevant organisations (ISPOR, SMDM, OHE, CRD etc) to establish a registry of this nature. If you don’t do it soon I’ll have to do it myself.
[…] used this blog in the past to propose more transparent research practices and to complain about publication practices in health economics generally, while […]
[…] It’s a big one to address, but it can be tackled in stages. I’ve written before that decision models should be published. This is a no-brainer, and I remain dumbfounded by the fact that funders don’t insist on […]
You make some good points Chris and as a journal editor model transparency and validation has concerned me for some time. Unfortunately, academic competition and (particularly) financial interest have corrupted the pursuit of science. I regularly encounter authors unwilling to share their models with reviewers let alone readers. This also happens with survey-based research where authors are often unwilling to publish the whole survey in case someone uses it without paying a licensing fee. Unfortunately, the public good doesn’t enter into the equation. The lure of consultancy fees is too great.
We discussed model transparency at length in the CHEERS task force (http://www.ispor.org/TaskForces/EconomicPubGuidelines.asp) and our recommendations are similar to those of Eddy et al. Jon Karnon has also written on the need for publically available reference case models.
I think the registry presents the opportunity to create a valuable public good. Health economics seems quite far ahead of other areas of economics in terms of cost-benefit analysis – a registry like this could be of great use in other areas where interventions and policies need to be assessed. Techniques like agent based modelling are becoming more popular in economics, for example, the failure or DSGE models in macroeconomics has led to a search for alternative methods. Health economists should be promoting their expertise as much as possible.