Background Systematic reviews (SRs) have become increasingly popular to a wide range of stakeholders. We set out to capture a representative cross-sectional sample of published SRs and examine them in terms of a broad range of epidemiological, descriptive, and reporting characteristics, including emerging aspects not previously examined. Methods and Findings We searched Medline for SRs indexed during November 2004 and written in English. Citations were screened and those meeting our inclusion criteria were retained. Data were collected using a 51-item data collection form designed to assess the epidemiological and reporting details and the bias-related aspects of the reviews. The data were analyzed descriptively. In total 300 SRs were identified, suggesting a current annual publication rate of about 2,500, involving more than 33,700 separate studies including one-third of a million participants. The majority (272 [90.7%]) of SRs were reported in specialty journals. Most reviews (213 [71.0%]) were categorized as therapeutic, and included a median of 16 studies involving 1,112 participants. Funding sources were not reported in more than one-third (122 [40.7%]) of the reviews. Reviews typically searched a median of three electronic databases and two other sources, although only about two-thirds (208 [69.3%]) of them reported the years searched. Most (197/295 [66.8%]) reviews reported information about quality assessment, while few (68/294 [23.1%]) reported assessing for publication bias. A little over half (161/300 [53.7%]) of the SRs reported combining their results statistically, of which most (147/161 [91.3%]) assessed for consistency across studies. Few (53 [17.7%]) SRs reported being updates of previously completed reviews. No review had a registration number. Only half (150 [50.0%]) of the reviews used the term “systematic review” or “meta-analysis” in the title or abstract. There were large differences between Cochrane reviews and non-Cochrane reviews in the quality of reporting several characteristics. Conclusions SRs are now produced in large numbers, and our data suggest that the quality of their reporting is inconsistent. This situation might be improved if more widely agreed upon evidence-based reporting guidelines were endorsed and adhered to by authors and journals. These results substantiate the view that readers should not accept SRs uncritically.
References
[1]
Canadian Institutes of Health Research. Available at: http://www.cihr-irsc.gc.ca/e/documents/r？ct_reg_e.pdf. Accessed 24 January 2006.
[2]
Young C, Horton R (2005) Putting clinical trials into context. Lancet 366: 107.
[3]
Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC (1987) Meta-analysis of randomized controlled trials. N Engl J Med 316: 450–455.
[4]
McAlister FA, Clark HD, van Walraven C, Straus SE, Lawson F, et al. (1999) The medical review article revisited: Has the science improved? Ann Intern Med 131: 947–951.
[5]
Mulrow CD (1987) The medical review article: State of the science. Ann Intern Med 106: 485–488.
[6]
Jadad AR, McQuay HJ (1996) Meta-analyses to evaluate analgesic interventions: A systematic qualitative review of their methodology. J Clin Epidemiol 49: 235–243.
[7]
Shea B, Moher D, Graham I, Pham BA, Tugwell P (2002) A comparison of the quality of Cochrane reviews and systematic reviews published in paper-based journals. Eval Health Prof 25: 116–129.
[8]
Jadad AR, Moher M, Browman GP, Booker L, Sigouin C, et al. (2000) Systematic reviews and meta-analyses on treatment of asthma: Critical evaluation. BMJ 320: 537–540.
[9]
Delaney A, Bagshaw SM, Ferland A, Manns B, Laupland KB, et al. (2005) A systematic evaluation of the quality of meta-analyses in the critical care literature. Crit Care 9: R575–R582.
[10]
Bhandari M, Morrow F, Kulkarni AV, Tornetta P (2001) Meta-analyses in orthopaedic surgery. A systematic review of their methodologies. J Bone Joint Surg Am 83A: 15–24.
[11]
Glenny AM, Esposito M, Coulthard P, Worthington HV (2003) The assessment of systematic reviews in dentistry. Eur J Oral Sci 111: 85–92.
[12]
Hayden JA, Cote P, Bombardier C (2006) Evaluation of the quality of prognosis studies in systematic reviews. Ann Inter Med 144: 427–437.
[13]
Whiting P, Rutjes AW, Dinnes J, Reitsma JB, Bossuyt PM, et al. (2005) A systematic review finds that diagnostic reviews fail to incorporate quality despite available tools. J Clin Epidemiol 58: 1–12.
[14]
Montori VM, Wilczynski NL, Morgan D, Haynes RB (2005) Optimal search strategies for retrieving systematic reviews from Medline analytical survey. BMJ 330: 68.
[15]
Fritsche L, Einecke G, Fleiner F, Dragun D, Neumayer HH, et al. (2004) Reports of large immunosuppression trials in kidney transplanation: Room for improvement. Am J Transplant 4: 738–743.
[16]
Mallett S, Clarke M (2002) The typical Cochrane review: How many trials, how many participants. Int J Technol Assess Health Care 18: 820–823.
[17]
Chan AW, Altman DG (2005) Epidemiology and reporting of randomised trials published in PubMed journals. Lancet 365: 1159–1162.
[18]
Moja LP, Telaro E, D'Amico R, Moschetti I, Coe L, et al. (2005) Assessment of methodological quality of primary studies by systematic reviews: Results of the metaquality cross sectional study. BMJ 330: 1053–1058.
[19]
Lexchin J, Bero LA, Djulbegovic B, Clark O (2003) Pharmaceutical industry sponsorship and research outcome and quality: Systematic review. BMJ 326: 1167–1170.
[20]
Dickersin K (2005) Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm. In: Rothstein HR, Sutton AJ, Borenstein M, editors. Publication Bias in Meta-Analysis—Prevention, Assessment and Adjustments. West Sussex (UK): John Wiley & Sons. 356 p.
[21]
Sutton AJ (2005) Evidence concerning the consequences of publication and related biases. In: Rothstein HR, Sutton AJ, Borenstein M, editors. Publication Bias in Meta-Analysis—Prevention, Assessment and Adjustments. West Sussex (UK): John Wiley & Sons. 356 p.
[22]
Silagy CA, Middleton P, Hopewell S (2002) Publishing protocols of systematic reviews: Comparing what was done to what was planned. JAMA 287: 2831–2834.
[23]
Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG (2004) Emperical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. JAMA 291: 2457–2465.
[24]
Chan AW, Krleza-Jeric K, Schmid I, Altman DG (2004) Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ 171: 735–740.
[25]
Moher D, Tsertsvadze A (2006) Systematic reviews: When is an update an update? Lancet 367: 881–883.
[26]
Simes RJ (1986) Publication bias: The case for an international registry of clinical trials. J Clin Oncol 4: 1529–1541.
[27]
Whittington CJ, Kendall T, Fonagy P, Cottrell D, Cotgrove A, et al. (2004) Selective serotonin reuptake inhibitors in childhood depression: Systematic review of published versus unpublished data. Lancet 363: 1341–1345.
[28]
De Angelis C, Drazan JM, Frizelle FA, Haug C, Hoey J, et al. (2004) International Committee Medical Journal Editors. Clinical trial registration: A statement from the International Committee of Medical Journal Editors. CMAJ 171: 606–607.
[29]
Centre for Reviews and Dissemination. Available at: http://www.york.ac.uk/inst/crd/listong.h？tm. Accessed 22 January 2006.
[30]
Australia The Joanna Briggs Institute. Available at: http://www.joannabriggs.edu.au/pubs/syst？ematic_reviews_prot.php. Accessed 22 January 2006.
[31]
Plint A, Moher D, Altman DG, Schulz KF, Morrison A (2006) Does the CONSORT checklist improve the quality of reports of randomized controlled trials? A systematic review. Med J Aust 185: 263–267.
[32]
Moher D, Schulz KF, Altman DG, for the CONSORT group (2001) The CONSORT statement: Revised recommendations for improving the quality of reports of parallel group randomized trials. Ann Intern Med 134: 657–662.
[33]
Rennie D (1997) How to report randomized controlled trials: The CONSORT statement. JAMA 276: 649.
[34]
Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, et al. (1999) Improving the quality of reporting of meta-analysis of randomized controlled trials: The QUOROM statement. Lancet 354: 1896–1900.
[35]
Delaney A, Bagshaw SM, Ferland A, Manns B, Laupland KB, et al. (2005) A systematic evaluation of the quality of meta-analyses in the critical care literature. Crit Care 9: R575–R582.
[36]
Montori VM, Wilczynski NL, Morgan D, Haynes RB (2005) Optimal search strategies for retrieving systematic reviews from Medline: Analytical survey. BMJ 330: 68.
[37]
Schulman J-LA, Nelson SJ (2005) Systematic reviews and medical vocabulary: Rapid response to Montori, Wilczynski, Morgan et al. BMJ eLetters. 330. 68#96833. Available at: http://www.bmj.com/cgi/eletters/330/7482？/68#96283. Accessed 25 February 2007.
