Do health technology assessments comply with QUOROM diagram guidance? An empirical study

We searched Medline to retrieve all systematic reviews of therapeutic interventions in the HTA monograph series published from 2001 to 2005. Two researchers recorded whether each study contained a meta-analysis of controlled trials, whether a QUOROM flow diagram was presented and, if so, whether it expressed the relationship between the number of citations and the number of studies. We used Cohen's kappa to test inter-rater reliability.87 systematic reviews were retrieved. There was good and excellent inter-rater reliability for, respectively, whether a review contained a meta-analysis and whether each diagram contained a citation-to-study relationship. 49% of systematic reviews used a study selection flow diagram. When only systematic reviews containing a meta-analysis were analysed, compliance was only 32%. Only 20 studies (23% of all systematic reviews; 43% of those having a study selection diagram) had a diagram which expressed the relationship between citations and studies.Compliance with the recommendations of the QUOROM statement is not universal in systematic reviews or meta-analyses. Flow diagrams make the conduct of study selection transparent only if the relationship between citations and studies is clearly expressed. Reviewers should understand what they are counting: citations, papers, studies and trials are fundamentally different concepts which should not be confused in a diagram.With an estimated seven systematic reviews being published every day [1], there is an increasing concern with the quality of reporting. Recognised standards exist for reporting meta-analyses of trials and observational studies [2,3] However there are no standards for systematic reviews in general (see terminology detailed in Table 1 for the distinction between systematic reviews, health technology assessments and meta-analyses). As a result, peer-reviewed publications frequently demand that standards devised specifically for reporting meta-analyses should inform the reporting