Decision models can be very useful. They can also be very bad. ISPOR and SMDM recently got together to form the Modeling Good Research Practices Task Force and have produced a set of reports. One of the reports, by David Eddy and colleagues, considers Model Transparency and Validation. The authors discuss the importance of transparency and validation in ensuring model credibility. They provide an excellent set of best practices that all modellers should follow. Go and read it.
The problem
The apparent effectiveness of a health care intervention can be distorted in many ways. Ben Goldacre‘s new book does a pretty good job of covering these. When it comes to cost-effectiveness the opportunities possibilities for distortion increase exponentially. A recent paper investigated FDA actions against health economic promotions. Many companies seek to promote products based on their economic value of “saving money” or “lowering costs”. The most common type of economic violation was an implied claim of cost savings due to work productivity or functioning. One can imagine how an unjustified figure of this nature could go unrecognised in a model.
There will always be intentionally misleading models, with which regulators will have to deal. However, most researchers’ models will have validity issues in spite of good intentions. If Eddy et al’s best practices were adhered to by good researchers then bad models could be easily identified and amended. The authors argue that, to ensure transparency, each model should be accompanied by both technical and non-technical documentation available to interested readers. However, a feasible process of ensuring transparency in this way does not currently exist.
The solution
The solution is to establish a model registry and database. It is unrealistic to expect journal publishers to accept extensive appendices, to which additions would need to be made as a model is developed.
Setting up a registry and online database would be very simple. Inclusion in the registry should require a minimum submission to the database: Eddy et al’s proposed non-technical documentation. Further information and files could be optional. Each model, whether single-application or multi-application, would be assigned a registration number and a corresponding database entry and webpage. The webpage could include all of the technical and non-technical documentation required by Eddy et al’s best practices. It could also include relevant citations, manuals and code. The site should provide a means for feedback and discussion. As models are updated and improved, or used in other applications, details can be added about subsequent versions. I’m not convinced of the argument presented by Eddy et al regarding intellectual property. This is meant to be science; one wouldn’t expect a health researcher to keep their trial design secret. Regardless, a registry would further protect intellectual property rights. Copyright is an automatic right in most countries, and plagiarism or use without appropriate attribution could easily be recognised and acted upon in a transparent system. The registry would provide a means of ensuring appropriate attribution of intellectual property.
There will be costs associated with maintaining such a registry and database (web hosting, doi fees), but most would be achieved by the contribution of researchers’ time. Academics are well accustomed to pro bono work and this shouldn’t represent a serious barrier. The primary challenge would be in encouraging researchers to submit their models to the registry. Model registration could not be a pre-requisite for publication in the same way as trial registration, so researchers would have to see other incentives. I would see the primary benefit as increased citations. If people are allowed to use existing models this will lead to further citation of the original, which could be facilitated by the provision of a Digital Object Identifier and would promote model validation.
An open and transparent registry would enable the research community to investigate the validity of models, which may be used to inform policy decisions. A registry would also help prevent the duplication of efforts. So, I call on relevant organisations (ISPOR, SMDM, OHE, CRD etc) to establish a registry of this nature. If you don’t do it soon I’ll have to do it myself.
