Background Treatment recommendations for the same condition from different guideline bodies often disagree, even when the same randomized controlled trial (RCT) evidence is cited. Guideline appraisal tools focus on methodology and quality of reporting, but not on the nature of the supporting evidence. This study was done to evaluate the quality of the evidence (based on consideration of its internal validity, clinical relevance, and applicability) underlying therapy recommendations in evidence-based clinical practice guidelines. Methods and Findings A cross-sectional analysis of cardiovascular risk management recommendations was performed for three different conditions (diabetes mellitus, dyslipidemia, and hypertension) from three pan-national guideline panels (from the United States, Canada, and Europe). Of the 338 treatment recommendations in these nine guidelines, 231 (68%) cited RCT evidence but only 105 (45%) of these RCT-based recommendations were based on high-quality evidence. RCT-based evidence was downgraded most often because of reservations about the applicability of the RCT to the populations specified in the guideline recommendation (64/126 cases, 51%) or because the RCT reported surrogate outcomes (59/126 cases, 47%). Conclusions The results of internally valid RCTs may not be applicable to the populations, interventions, or outcomes specified in a guideline recommendation and therefore should not always be assumed to provide high-quality evidence for therapy recommendations.
References
[1]
McMurray J, Swedberg K (2006) Treatment of chronic heart failure: A comparison between the major guidelines. Eur Heart J 27: 1773–1777.
[2]
McAlister FA, Campbell NRC, Zarnke K, Levine M, Graham I (2001) The management of hypertension in Canada: A review of current guidelines, their shortcomings, and implications for the future. CMAJ 164: 517–522.
[3]
GRADE Working Group (2004) Grading quality of evidence and strength of recommendations. BMJ 328: 1490.
[4]
Grol R, Dalhuijsen J, Thomas S, in't Veld C, Rutten G, et al. (1998) Attributes of clinical guidelines that influence use of guidelines in general practice: Observational study. BMJ 317: 858–861.
[5]
Burgers JS (2006) Guideline quality and guideline content: Are they related? Clin Chem 52: 3–4.
[6]
Vlayen J, Aertgeerts B, Hannes K, Sermeus W, Ramaekers D (2005) A systematic review of appraisal tools for clinical practice guidelines: Multiple similarities and one common deficit. Int J Qual Health Care 17: 235–242.
[7]
Burgers JS, Vailey JV, Klazinga NS, van der Bij AK, Grol R, et al. (2002) Inside guidelines. Comparative analysis of recommendations and evidence in diabetes guidelines from 13 countries. Diabetes Care 25: 1933–1939.
[8]
Atkins D, Eccles M, Flottorp S, Guyatt GH, Henry D, et al. (2004) Systems for grading the quality of evidence and the strength of recommendations I: Critical appraisal of existing approaches. BMC Health Services Research 4: 38.
[9]
McAlister FA (2006) The Canadian Hypertension Education Program (CHEP)—A Unique Canadian Initiative. Can J Cardiol 22: 559–564.
[10]
Harris SB, Capes SE, Lillie D, Lank CN, Mahon J, et al. (2003) Canadian Diabetes Association 2003 clinical practice guidelines for the prevention and management of diabetes in Canada. Can J Diabetes 27: S1–S152.
[11]
American Diabetes Association (2005) Standards of medical care in diabetes. Diabetes Care 28(Suppl 1): S4–36.
[12]
International Diabetes Federation (2005) Clinical guidelines task force: Global guidelines for type 2 diabetes. International Diabetes Federation website. Available: http://www.idf.org/home/index.cfm?node=1？457. Accessed 11 December 2006.
[13]
Genest J, Frohlich J, Fodor G, McPherson R (2003) Recommendations for the management of dyslipidemia and the prevention of cardiovascular disease. CMAJ 169: 921–924.
[14]
Grundy SM, Becker D, Clark LT, Cooper RS, Denke MA, et al. (2001) Executive summary of the third report of the National Cholesterol Education Program (NCEP) expert panel on detection, evaluation, and treatment of high blood cholesterol in adults. National Heart Lung and Blood Institute Website. Available: http://www.nhlbi.nih.gov. Accessed 11 December 2006.
[15]
De Backer G, Ambrosioni E, Borch-Johnsen K, Brotons C, Cifkova R, et al. (2003) European guidelines on cardiovascular disease prevention in clinical practice: Third joint task force of European and other societies on cardiovascular disease prevention in clinical practice. Eur Heart J 24: 1601–1610.
[16]
Hemmelgarn B, McAlister FA, Grover S, Myers MG, McKay DW, et al. (2006) The 2006 Canadian Hypertension Education Program (CHEP) recommendations for the management of hypertension: Part I, Blood pressure measurement, diagnosis, and assessment of risk. Can J Cardiol 22: 573–581.
[17]
Khan NA, McAlister FA, Rabkin SW, Padwal R, Feldman RD, et al. (2006) The 2006 Canadian Hypertension Education Program (CHEP) recommendations for the management of hypertension: Part 2, Therapy. Can J Cardiol 22: 583–593.
[18]
Chobanian AV, Bakris GL, Black HR, Cushman WC, Green LA, et al. (2003) The seventh report of the Joint National Committee on prevention, detection, evaluation, and treatment of high blood pressure. The JNC 7 Report. JAMA 289: 2560–2572.
[19]
Guidelines Committee (2003) 2003 European Society of Hypertension–European Society of Cardiology guidelines for the management of arterial hypertension. J Hypertens 21: 1011–1053.
[20]
Cluzeau FA, Littlejohns P, Grimshaw JM, Feder G, Moran SE (1999) Development and application of a generic methodology to assess the quality of clinical guidelines. Int J Qual Health Care 11: 21–28.
[21]
Guyatt GH, Rennie D, editors. (2002) Users' guides to the medical literature. A manual for evidence-based clinical practice. Chicago: American Medical Association. 705 p.
[22]
Psaty BM, Lumley T, Furberg CD, Schellenbaum G, Pahor M, et al. (2003) Health outcomes associated with various antihypertensive therapies used as first-line agents: A network meta-analysis. JAMA 289: 2534–2544.
[23]
Pitt B, Zannad F, Remme WJ, Cody R, Castaigne A, et al. (1999) The effect of spironolactone on morbidity and mortality in patients with severe heart failure. N Engl J Med 341: 709–717.
[24]
Abramson J, Wright JM (2007) Are lipid-lowering guidelines evidence-based? Lancet 369: 168–169.
[25]
Eknoyan G, Levin A, Levin N, Kasiske B, Cosio FG, et al. (2003) K/DOQI clinical practice guidelines for managing dyslipidemias in chronic kidney disease. Am J Kidney Dis 41(Suppl 3): S1–S91. for the National Kidney Foundation.
[26]
Heart Protection Study Collaborative Group (2002) MRC/BHF Heart Protection Study of cholesterol lowering with simvastatin in 20 536 high-risk individuals: A randomised placebo-controlled trial. Lancet 360: 7–22.
[27]
Wanner C, Krane V, Marz W, Olschewski M, Mann JF, et al. (2005) Atorvastatin in patients with type 2 diabetes mellitus undergoing hemodialysis. N Engl J Med 353: 238–248.
[28]
Rothwell PM (2005) External validity of randomised controlled trials: “To whom do the results of this trial apply?” Lancet 365: 82–93.
[29]
Burgers JS, van Everdingen JJE (2004) Beyond the evidence in clinical guidelines. Lancet 364: 392–393.
[30]
Raine R, Sanderson C, Hutchings A, Carter S, Larkin K, et al. (2004) An experimental study of determinants of group judgements in clinical guideline development. Lancet 364: 429–437.
[31]
Pagliari C, Grimshaw J, Eccles M (2001) The potential influence of small group processes on guideline development. J Eval Clin Pract 7: 165–173.
[32]
Shaneyfelt TM, Mayo-Smith MF, Rothwangl J (1999) Are guidelines following guidelines? The methodological quality of clinical practice guidelines in the peer-reviewed medical literature. JAMA 281: 1900–1905.
[33]
Graham ID, Beardall S, Carter AO, Glennie J, Hebert P, et al. (2001) What is the quality of drug therapy clinical practice guidelines in Canada? CMAJ 165: 157–163.
[34]
Grilli R, Magrini N, Penna A, Mura G, Liberati A (2000) Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet 355: 103–106.
[35]
Vincent JL (2006) Is the current management of severe sepsis and septic shock really evidence based? PLoS Med 3: e346.. doi:10.1371/journal.pmed.0030346.
[36]
Grant J, Cotttrell R, Cluzeau F, Fawcett G (2000) Evaluating “payback” on biomedical research from papers cited in clinical guidelines: Applied bibliometric study. BMJ 320: 1107–1111.
[37]
Ramsay LE, Wallis EJ, Yeo WW, Jackson PR (1998) The rationale for differing national recommendations for the treatment of hypertension. Am J Hypertens 11: 79S–88S.
[38]
Schünemann HJ, Hill SR, Kakad M, Vist GE, Bellamy R, et al. (2007) Transparent development of the WHO rapid advice guidelines. PLoS Med 4: e119. doi:10.1371/journal.pmed.0040119.
[39]
Persaud N, Mamdani MM (2006) External validity: The neglected dimension in evidence ranking. J Eval Clin Practice 12: 450–453.
[40]
Rubins HB, Robins SJ, Collins D, Fye CL, Anderson JW, et al. (1999) Gemfibrozil for the secondary prevention of coronary heart disease in men with low levels of high-density lipoprotein cholesterol. N Engl J Med 341: 410–418.
[41]
Sacks FM, Svetkey LP, Vollmer WM, Appel LJ, Bray GA, et al. (2001) Effects on blood pressure of reduced dietary sodium and the Dietary Approaches to Stop Hypertension (DASH) diet. N Engl J Med 344: 3–10.
[42]
Nakao N, Yoshimura A, Morita H, Takada M, Kayano T, et al. (2003) Combination treatment of angiotensin-II receptor blocker and angiotensin-converting-enzyme inhibitor in non-diabetic renal disease (COOPERATE): A randomized controlled trial. Lancet 361: 117–124.
[43]
Antithrombotic Trialists' Collaboration (2002) Collaborative meta-analysis of randomized trials of antiplatelet therapy for prevention of death, myocardial infarction, and stroke in high risk patients. BMJ 324: 71–86.
[44]
Kjeldsen SE, Dahl？f B, Devereux RB, Julius S, Aurup P, et al. (2002) Effects of losartan on cardiovascular morbidity and mortality in patients with isolated systolic hypertension and left ventricular hypertrophy: A Losartan Intervention for Endpoint Reduction (LIFE) substudy. JAMA 288: 1491–1498.
[45]
UK Prospective Diabetes Study Group (1998) Efficacy of atenolol and captopril in reducing risk of macrovascular and microvascular complications in type 2 diabetes (UKPDS 39). BMJ 317: 713–720.
