Background Peer review is considered crucial to the selection and publication of quality science, but very little is known about the previous experiences and training that might identify high-quality peer reviewers. The reviewer selection processes of most journals, and thus the qualifications of their reviewers, are ill defined. More objective selection of peer reviewers might improve the journal peer review process and thus the quality of published science. Methods and Findings 306 experienced reviewers (71% of all those associated with a specialty journal) completed a survey of past training and experiences postulated to improve peer review skills. Reviewers performed 2,856 reviews of 1,484 separate manuscripts during a four-year study period, all prospectively rated on a standardized quality scale by editors. Multivariable analysis revealed that most variables, including academic rank, formal training in critical appraisal or statistics, or status as principal investigator of a grant, failed to predict performance of higher-quality reviews. The only significant predictors of quality were working in a university-operated hospital versus other teaching environment and relative youth (under ten years of experience after finishing training). Being on an editorial board and doing formal grant (study section) review were each predictors for only one of our two comparisons. However, the predictive power of all variables was weak. Conclusions Our study confirms that there are no easily identifiable types of formal training or experience that predict reviewer performance. Skill in scientific peer review may be as ill defined and hard to impart as is “common sense.” Without a better understanding of those skills, it seems unlikely journals and editors will be successful in systematically improving their selection of reviewers. This inability to predict performance makes it imperative that all but the smallest journals implement routine review ratings systems to routinely monitor the quality of their reviews (and thus the quality of the science they publish).
References
[1]
Schulman K, Sulmasy DP, Roney D (1994) Ethics, economics, and the publication policies of major medical journals. JAMA 272: 154–156.
[2]
Callaham M (2003) The evaluation and training of peer reviewers. In: Godlee F, Jefferson T, editors. Peer review in health sciences. 2nd Ed. London: BMJ Books. pp. 164–182.
[3]
Stossel TP (1985) Reviewer status and review quality: Experience of the Journal of Clinical Investigation. N Engl J Med 312: 658–659.
[4]
Evans AT, McNutt RA, Fletcher SW, Fletcher RH (1993) The characteristics of peer reviewers who produce good-quality reviews. J Gen Intern Med 8: 422–428.
[5]
Black N, van Rooyen S, Godlee F, Smith R, Evans S (1998) What makes a good reviewer and a good review for a general medical journal? JAMA 280: 231–233.
[6]
Kliewer MA, Freed KS, DeLong DM, Pickhardt PJ, Provenzale JM (2005) . Reviewing the reviewers. Comparison of review quality and reviewer characteristics at the Am J Roentgenol. 184. : 1731–1735.
[7]
Journal Citation Reports (2004) ISI Web of Knowledge. The Thomson Corporation. Available: http://www.ama-assn.org/public/peer/abst？racts.html. Accessed: 2/15/06.
[8]
Callaham ML, Baxt WG, Waeckerle JF, Wears RL (1998) Reliability of editors' subjective quality ratings of peer reviews of manuscripts. JAMA 280: 229–231.
[9]
van Rooyen S, Black N, Godlee F (1999) Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol 52: 625–629.
[10]
Baxt WG, Waeckerle JF, Berlin JA, Callaham ML (1998) Who reviews the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer performance. Ann Emerg Med 32: 310–317.
[11]
Nylenna M, Riis P, Karlsson Y (1994) Multiple blinded reviews of the same two manuscripts. Effects of referee characteristics and publication language. JAMA 272: 149–151.
[12]
Schroter S, Black N, Evans S, Carpenter J, Godlee F, et al. (2004) Effects of training on quality of peer review: Randomised controlled trial. BMJ 328: 673–675.
[13]
Callaham M (1997) Training programs for peer reviewers. CBE Views (Council of Biology Editors) 20: 192–193.
[14]
Callaham ML, Schriger DL (2002) Effect of structured workshop training on subsequent performance of journal peer reviewers. Ann Emerg Med 40: 323–328.
[15]
Callaham ML, Wears RL, Waeckerle JF (1998) Effect of attendance at a training session on peer reviewer quality and performance. Ann Emerg Med 32: 318–322.
[16]
Callaham ML, Knopp RK, Gallagher EJ (2002) Effect of written feedback by editors on quality of reviews: Two randomized trials. JAMA 287: 2781–2783.
[17]
Klein G (1998) Sources of power: How people make decisions. Cambridge (Massachusetts): MIT Press. 330 p.
[18]
Steinmann WC, Lebeau DL, Michaels RK (1997) A survey of journal editors regarding the review process for original clinical research [abstract]. JAMA and BMJ, sponsors.
[19]
Kliewer MA, DeLong DM, Freed K, Jenkins CB, Paulson EK, et al. (2004) Peer review at the American Journal of Roentgenology: How reviewer and manuscript characteristics affected editorial decisions on 196 major papers. Am J Roentgenol 183: 1545–1550.
