Editorial policies between the top medical journals differ. Some take a ‘crusading’ view and campaign on contemporary health issues. The BMJ falls into this camp, although this has sometimes led them to take political positions that might be contrary to the evidence. Nevertheless, the editorial agenda of the BMJ is clear, readers know what they are backing. The NEJM on the other hand seems to have adopted a more opaque position.
On the face of it the NEJM seems to support a position of ‘if a randomised controlled trial (RCT) has been conducted and it’s published then that’s the last word on the matter’. Some recent examples illustrate this. Ben Goldacre and colleagues in the COMPARE project received a dismissal of their letters penned to the NEJM that expressed concerns over trials that had not reported on the primary outcomes specified in their protocols or reported different outcomes. The New York Times reports on potential flaws or even misconduct in a mega trial of Xarelto, an anticlotting drug, for which the manufacturers are currently being sued. The NEJM, which published the trial, dismissed the relevance of the claims and defended the trial. And, in a recent, controversial editorial, the NEJM appears to endorse the view that researchers who re-analyse trial data from other studies are ‘research parasites’.
This view is not unique to the NEJM. It reflects a broader view that RCTs are definitive and research in top impact factor journals more so. But, scientists are fallible, and RCTs can be flawed and present biased results. For example, in a study of the top four medical journals 95% of RCTs had some missing data, with a median percentage of 9% dropout, and in many cases adequate missing data methods were not used. Publication should not be the final stage of a piece of research but part of an ongoing process.
Part of the problem may lie with the false dichotomy imposed by hypothesis testing and statistical significance. A treatment either works or does not work or it is safe or it is not safe. But, for the most part, these tests are based solely on asking whether the data are compatible with the hypothesis or whether it’s unlikely. All the other forms of uncertainty are not taken into account such as missing data, a lack of adequate allocation concealment, or a lack of double blinding. The researchers could have chosen any number of different tests or comparisons and the choice could be contingent on the data, potentially rendering the p-value meaningless.
Results from RCTs are used to make important clinical and policy decisions. Scrutiny and debate are essential to ensure that the best decisions are made. This includes allowing for an appropriate representation of the uncertainty surrounding a decision. The trust endowed by a high impact factor should bring a responsibility to ensure that well founded critical or dissenting views on published research are appropriately represented. RCTs should be subject to as much scrutiny as any other form of research. Vioxx should serve as an important reminded of this.