Cohen’s kappa is a widely used association coefficient for summarizing interrater agreement on a nominal scale. Kappa reduces the ratings of the two observers to a single number. With three or more categories it is more informative to summarize the ratings by category coefficients that describe the information for each category separately. Examples of category coefficients are the sensitivity or specificity of a category or the Bloch-Kraemer weighted kappa. However, in many research studies one is often only interested in a single overall number that roughly summarizes the agreement. It is shown that both the overall observed agreement and Cohen’s kappa are weighted averages of various category coefficients and thus can be used to summarize these category coefficients. 1. Introduction In various fields of science it is frequently required that an observer classifies a set of subjects into three or more nominal categories that are defined in advance. The observer may be a clinician who classifies children on the severity of a disease, a pathologist that rates the severity of lesions from scans, or a coder that transcribes interviews. If the observer did not fully understand what he or she was asked to interpret, or if the definition of the categories is ambiguous, the reliability of the rating system is at stake. To assess the reliability of the system researchers typically ask two or more observers to rate the same set of subjects independently. An analysis of the agreement between the observers can then be used as an indicator of the quality of the category definitions and the raters’ ability to apply them. High agreement between the ratings would indicate consensus in the diagnosis and interchangeability of the ratings. There are several association coefficients that can be used for summarizing agreement between two observers [1–3]. In biomedical and behavioral science research the most widely used coefficient for summarizing agreement on a scale with two or more nominal categories is Cohen’s kappa [4–8]. The coefficient has been applied in thousand of research studies and is also frequently used for summarizing agreement if we have observers of one type paired with observers of a second type, and each of the observers assigns a subject to one of categories. A closely related coefficient is Scott’s pi [9]. The latter coefficient is commonly used in the field of content analysis [2, 10]. The two coefficients have similar formulas and differ in how agreement under chance is defined [3, 11]. Cohen’s kappa reduces the ratings of the two observers to a
References
[1]
L. M. Hsu and R. Field, “Interrater agreement measures: comments on Kappan, Cohen's Kappa, Scott's , and Aickin's ,” Understanding Statistics, vol. 2, pp. 205–219, 2003.
[2]
K. Krippendorff, “Reliability in content analysis: some common misconceptions and recommendations,” Human Communication Research, vol. 30, no. 3, pp. 411–433, 2004.
[3]
M. J. Warrens, “Inequalities between kappa and kappa-like statistics for K？x？K tables,” Psychometrika, vol. 75, no. 1, pp. 176–185, 2010.
[4]
J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement, vol. 20, pp. 37–46, 1960.
[5]
J. A. Hanley, “Standard error of the kappa statistic,” Psychological Bulletin, vol. 102, no. 2, pp. 315–321, 1987.
[6]
M. Maclure and W. C. Willett, “Misinterpretation and misuse of the Kappa statistic,” American Journal of Epidemiology, vol. 126, no. 2, pp. 161–169, 1987.
[7]
M. J. Warrens, “On the equivalence of Cohen's kappa and the Hubert-Arabie adjusted Rand index,” Journal of Classification, vol. 25, no. 2, pp. 177–183, 2008.
[8]
M. J. Warrens, “Cohen's kappa can always be increased and decreased by combining categories,” Statistical Methodology, vol. 7, no. 6, pp. 673–677, 2010.
[9]
W. A. Scott, “Reliability of content analysis: the case of nominal scale coding,” Public Opinion Quarterly, vol. 19, no. 3, pp. 321–325, 1955.
[10]
K. Krippendorff, Content Analysis: An Introduction to Its Methodology, Sage, Thousands Oaks, Calif, USA, 2nd edition, 2004.
[11]
K. Krippendorff, “Association, agreement, and equity,” Quality and Quantity, vol. 21, no. 2, pp. 109–123, 1987.
[12]
J. L. Fleiss, “Measuring agreement between two judges on the presence or absence of a trait,” Biometrics, vol. 31, no. 3, pp. 651–659, 1975.
[13]
J. L. Fleiss, Statistical Methods for Rates and Proportions, Wiley, New York, NY, USA, 1981.
[14]
J. L. Fleiss, B. Levin, and M. C. Paik, Statistical Methods for Rates and Proportions, John Wiley & Sons, New York, NY, USA, 3rd edition, 2003.
[15]
H. C. Kraemer, “Ramifications of a population model for κ as a coefficient of reliability,” Psychometrika, vol. 44, no. 4, pp. 461–472, 1979.
[16]
S. Vanbelle and A. Albert, “Agreement between two independent groups of raters,” Psychometrika, vol. 74, no. 3, pp. 477–491, 2009.
[17]
M. J. Warrens, “Cohen's kappa is a weighted average,” Statistical Methodology, vol. 8, no. 6, pp. 473–484, 2011.
[18]
H. C. Kraemer, V. S. Periyakoil, and A. Noda, “Kappa coefficients in medical research,” Statistics in Medicine, vol. 21, no. 14, pp. 2109–2129, 2002.
[19]
A. Agresti, “Modelling patterns of agreement and disagreement,” Statistical Methods in Medical Research, vol. 1, no. 2, pp. 201–218, 1992.
[20]
S. Fennig, T. J. Craig, M. Tanenberg-Karant, and E. J. Bromet, “Comparison of facility and research diagnoses in first-admission psychotic patients,” The American Journal of Psychiatry, vol. 151, no. 10, pp. 1423–1429, 1994.
[21]
F. B. Baulieu, “A classification of presence/absence based dissimilarity coefficients,” Journal of Classification, vol. 6, no. 2, pp. 233–246, 1989.
[22]
M. J. Warrens, “On similarity coefficients for tables and correction for chance,” Psychometrika, vol. 73, no. 3, pp. 487–502, 2008.
[23]
M. J. Warrens, “On association coefficients for tables and properties that do not depend on the marginal distributions,” Psychometrika, vol. 73, no. 4, pp. 777–789, 2008.
[24]
M. J. Warrens, “Chance-corrected measures for 2？×？2 tables that coincide with weighted kappa,” British Journal of Mathematical and Statistical Psychology, vol. 64, no. 2, pp. 355–365, 2011.
[25]
L. R. Dice, “Measures of the amount of ecologic association between species,” Ecology, vol. 26, pp. 297–302, 1945.
[26]
D. A. Bloch and H. C. Kraemer, “2？x？2 kappa coefficients: measures of agreement or association,” Biometrics, vol. 45, no. 1, pp. 269–287, 1989.
[27]
J. Cohen, “Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit,” Psychological Bulletin, vol. 70, no. 4, pp. 213–220, 1968.
[28]
M. J. Warrens, “Some paradoxical results for the quadratically weighted kappa,” Psychometrika, vol. 77, no. 2, pp. 315–323, 2012.
[29]
M. J. Warrens, “Equivalences of weighted kappas for multiple raters,” Statistical Methodology, vol. 9, no. 3, pp. 407–422, 2012.
[30]
M. J. Warrens, “Conditional inequalities between Cohen's kappa and weighted kappas,” Statistical Methodology, vol. 10, pp. 14–22, 2013.
[31]
J. S. Coleman, “Measures of concordance or consensus between members of social groups,” Johns Hopkins University, 1966.
[32]
R. J. Light, “Measures of response agreement for qualitative data: some generalizations and alternatives,” Psychological Bulletin, vol. 76, no. 5, pp. 365–377, 1971.
[33]
Y. M. M. Bishop, S. E. Fienberg, and P. W. Holland, Discrete Multivariate Analysis: Theory and Practice, MIT Press, Cambridge, UK, 1975.
[34]
A. Agresti, Categorical Data Analysis, Wiley Series in Probability and Statistics, Wiley-Interscience, Hoboken, NJ, USA, 2nd edition, 2002.
[35]
A. N. Albatineh, M. Niewiadomska-Bugaj, and D. Mihalko, “On similarity indices and correction for chance agreement,” Journal of Classification, vol. 23, no. 2, pp. 301–313, 2006.
[36]
J. L. Fleiss, J. Cohen, and B. S. Everitt, “Large sample standard errors of kappa and weighted kappa,” Psychological Bulletin, vol. 72, no. 5, pp. 323–327, 1969.
