There is now convincing evidence that the manufacturers of drugs are absolutely unscrupulous in the way in which their products are marketed. Invariably the benefits are emphasised and overplayed while any side-effects are minimized. However all drugs have to be thoroughly tested and evaluated before they can be granted approval by the authorities. But questions have been raised about the objectivity and reliability of the information submitted by the companies as part of the approval process and how this is communicated to the medical/scientific community.
Turner and colleagues examined the data submitted by drug companies to the US Food and Drugs Administration (FDA) in support of applications for approval for 12 different anti-depressants. This information was then compared with what was published in the scientific literature, which is how the medical community learns about the drugs (1). The authors had access to the official reviews conducted by the FDA and used this to classify the anti-depressants as positive, negative or questionable (neither positive nor negative). In total there were 74 studies submitted to the FDA, of which 38 were judged to be positive and 37 of these were subsequently published as scientific papers. This left 36, of which 24 were considered negative and 12 questionable. Three of these were published as not positive and 22 were not published. However the remaining 11 were published but in the opinion of Turner et al were presented as positive, despite the fact that this was in direct conflict to the FDA decision!
It was noted that in all these journal articles the methods reported appeared to depart from the pre-specified method submitted to the FDA. Although for each of these studies the finding with respect to the protocol-specified primary outcome was non-significant, invariably the authors were able to identify a different outcome that was positive which was then highlighted as if it were the primary outcome. The non-significant results for the pre-specified primary outcomes were either subordinated to non-primary positive results or omitted.
Essentially what is happening is that the positive results were made available which is perfectly reasonable. Some of the negative results probably never see the light of day but would certainly be of value to the medical profession. The twisting of results to give a favourable impression, which is in direct conflict with the results of the trial, is totally unacceptable.
A similar but more comprehensive approach has been adopted in 2 reports published by the Public Library of Science (PLoS). This is a non-profit scientific publishing project aimed at creating a library of open access journals and other scientific literature under an open content licence (2,3).
In the first one, all New Drug Applications (NDAs) submitted to the FDA between January 1998 and December 2000 were analysed (2). This consisted of 909 trials with 90 different drugs. Of the trials, 394 of them were matched to publications in the medical literature. A total of 340 trials were judged to be pivotal and 76% of these were published. There was strong evidence of publication bias in the sense that trials with statistically significant results were more likely to be published than those which were non-significant.
In the second study, Kristin Rising and colleagues analysed NDAs which were submitted in the years 2001 and 2002 (2). These were then compared with the relevant scientific publications. This study reached conclusions that were effectively the same as those studies described above. There was evidence of failure to publish and selective reporting of data. Once again the results of trials which were favourable to a specific drug were highly likely to be published. However where this was not the case it was found that there could be quite critical differences between the trials as they were described in the FDA reviews and their corresponding publications. There were 43 outcomes that did not favour the drug. Of these just under half of them (20) were not mentioned in the papers. In the remaining 23, the statistical significance changed between the NDA and the paper in 5 of them and in another 4 a positive outcome was given in the paper even though this was in conflict with the FDA review. In other words the favourable impression presented for these drugs could not possibly be justified.
It is clear that in the immediate period after approval by the FDA, which is the most relevant to public health, there is incomplete and selective publication of the results of the investigations which underpin the approval. As a consequence clinicians are likely to make inappropriate treatment decisions. The failure to provide complete and accurate information must be a severe impediment to any scientist wishing to improve the efficacy and safety of these drugs. In fact it has been advocated that publication bias should be regarded as a form of scientific misconduct (4).
A review by the Cochrane Collaboration identified 5 separate papers on publication bias which were published between 1992 and 1998 (5). These studies showed that trials with positive findings (defined either as those that were statistically significant, or those findings perceived to be important or striking, or those indicating a positive direction of treatment effect), had nearly four times the odds of being published compared to findings that were not statistically significant, or perceived as unimportant, or showing a negative or null direction of treatment effect. It was recommended that clinical trials are registered before recruiting participants so that review authors know about all potentially eligible studies, regardless of their findings.
In 2007 new legislation was introduced in Europe and in the US which has attempted to overcome these abuses. Nevertheless any meta-analysis which includes results obtained before that date must be regarded as suspect unless the authors were successful in obtaining all the relevant information on trials which produced negative outcomes that were not published.
REFERENCES
- E H Turner et el (2008) New England Journal of Medicine 358 (3) pp252-260
- K Lee et al (2008) PLoS Med 5 (9) e191
- K Rising et al (2008) PLoS Med 5 (11) e217
- I Chalmers (1990) Journal of the American Medical Association 263 (10) pp1405-1408
- S Hopewell et al (2009) The Cochrane Library, Issue 1: John Wiley & Sons