Reliability studies for coding contributing factors of incident reports in high hazard industries are rarely conducted and reported. Although the Human Factors Analysis and Classification System (HFACS) appears to have a larger number of such studies completed than most other systems doubt exists as the accuracy and comparability of results between studies due to aspects of methodology and reporting. This paper reports on a trial conducted on HFACS to determine its reliability in the context of military air traffic control (ATC). Two groups participated in the trial: one group comprised of specialists in the field of human factors, and the other group comprised air traffic controllers. All participants were given standardized training via a self-paced workbook and then read 14 incident reports and coded the associated findings. The results show similarly low consensus for both groups of participants. Several reasons for the results are proposed associated with the HFACS model, the context within which incident reporting occurs in real organizations and the conduct of the studies. 1. Introduction There are numerous techniques available for the classification of incident and accident contributing factors into codes that are fundamental to trend analyses and the mitigation of human error (e.g., TRACEr, [1]; SECAS, [2]; HFACS, [3]). Many of these techniques are in the form of taxonomies, containing separate categories and codes from which coders select and then apply to incident contributing factors. However, few taxonomies have been subject to independent reliability studies to provide evidence that the classification system can provide consistent coding over time and consensus amongst different coders. Such evidence is important because contributions to trend analysis are made via incident reports investigated by different safety investigators or analysts, often from different departments within the one organisation. The accuracy of the resultant analyses, which are key to the development of accident prevention measures, is therefore dependent on the ability of those contributors to achieve consensus on their classification decisions across all contributing factors highlighted in the reports [4]. Ross et al. [5] drew attention to the fact that reliability studies of taxonomies can be overlooked, use inappropriate methods, and be reported ambiguously. These faults are indeed prevalent in the relatively few reliability studies conducted on incident classification systems. For example, there is often inadequate reporting of the methodology in cited
References
[1]
S. T. Shorrock and B. Kirwan, “Development and application of a human error identification tool for air traffic control,” Applied Ergonomics, vol. 33, no. 4, pp. 319–336, 2002.
[2]
B. Wallace, A. Ross, J. Davies, L. Wright, and M. White, “The creation of a new minor event coding system,” Cognition, Technology and Work, vol. 4, no. 1, pp. 1–8, 2002.
[3]
D. Wiegmann and S. Shappell, A Human Error Approach to Aviation Accident Analysis: The Human Factors Analysis and Classification System, Ashgate Publishing, Aldershot, UK, 2003.
[4]
B. Wallace and A. Ross, Beyond Human Error: Taxonomies and Safety Science, CRC Press, Boca Raton, Fla, USA, 2006.
[5]
A. J. Ross, B. Wallace, and J. B. Davies, “Technical note: measurement issues in taxonomic reliability,” Safety Science, vol. 42, no. 8, pp. 771–778, 2004.
[6]
A. W. ElBardissi, D. A. Wiegmann, J. A. Dearani, R. C. Daly, and T. M. Sundt, “Application of the human factors analysis and classification system methodology to the cardiovascular surgery operating room,” Annals of Thoracic Surgery, vol. 83, no. 4, pp. 1412–1419, 2007.
[7]
A. Pape, D. Wiegmann, and S. Shappell, “Air Traffic Control (ATC) related accidents and incidents: a human factors analysis,” in Proceedings of the 11th International Symposium on Aviation Psychology, The Ohio State University, Columbus, Ohio, 2001.
[8]
J. Pounds, A. Scarborough, and S. Shappell, “A human factors analysis of air traffic control operational errors,” Aviation Space and Environmental Medicine, vol. 71, pp. 329–332, 2000.
[9]
S. Shappell, C. Detwiler, K. Holcomb, C. Hackworth, A. Boquet, and D. A. Wiegmann, “Human error and commercial aviation accidents: an analysis using the human factors analysis and classification system,” Human Factors, vol. 49, no. 2, pp. 227–242, 2007.
[10]
D. Wiegmann, T. Faaborg, A. Boquet, C. Detwiler, K. Halcomb, and S. Shappell, “Human error and general aviation accidents: a comprehensive, fine-grained analysis using HFACS,” Department of Transportation, FAA Report, DOT/FAA/AM-05/24, 2005.
[11]
S. Shappell and D. Wiegmann, “A human error analysis of general aviation controlled flight into terrain accidents occurring between 1990–1998,” Department of Transportation, FAA Report, DOT/FAA/AM-03/4, 2003.
[12]
J. Davies, A. Ross, B. Wallace, and L. Wright, Safety Management: A Qualitative Systems Approach, Taylor and Francis, London, UK, 2003.
[13]
J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement, vol. 20, no. 1, pp. 37–46, 1960.
[14]
W. M. Grove, N. C. Andreasen, P. McDonald-Scott, M. Keller, and R. Shapiro, “Reliability studies of psychiatric diagnosis. Theory and practice,” Archives of General Psychiatry, vol. 38, no. 4, pp. 408–413, 1981.
[15]
A. E. Maxwell, “Coefficients of agreement between observers and their interpretation,” British Journal of Psychiatry, vol. 130, no. 1, pp. 79–83, 1977.
[16]
E. L. Spitznagel and J. E. Helzer, “A proposed solution to the base rate problem in the kappa statistic,” Archives of General Psychiatry, vol. 42, no. 7, pp. 725–728, 1985.
[17]
N. A. Stanton and S. V. Steverage, “Learning to predict human error: issues of acceptability, reliability and validity,” Ergonomics, vol. 41, no. 11, pp. 1737–1756, 1998.
[18]
H. W. Wallén-Warner and J. Sandin, “The intercoder agreement when using the driving reliability and error analysis method in road traffic accident investigations,” Safety Science, vol. 48, no. 5, pp. 527–536, 2010.
[19]
S. Shappell and D. Wiegmann, “Applying Reason: the human factors analysis and classification system (HFACS),” Human Factors and Aerospace Safety, vol. 1, no. 1, pp. 59–86, 2001.
[20]
W. Li and D. Harris, “HFACS analysis of ROC Air Force aviation accidents: reliability analysis and crosscultural comparison,” International Journal of Applied Aviation Studies, vol. 5, no. 1, pp. 65–72, 2005.
[21]
S. W. J. Kozlowski and K. Hattrup, “A disagreement about within-group agreement: disentangling issues of consistency versus consensus,” Journal of Applied Psychology, vol. 77, no. 2, pp. 161–167, 1992.
[22]
P. Martin and P. Bateson, Measuring Behaviour: An Introductory Guide, Cambridge University Press, Cambridge, UK, 1993.
[23]
P. O'Connor, “HFACS with an additional layer of granularity: validity and utility in accident analysis,” Aviation Space and Environmental Medicine, vol. 79, no. 6, pp. 599–606, 2008.
[24]
N. S. Olsen and S. T. Shorrock, “Evaluation of the HFACS-ADF safety classification system: inter-coder consensus and intra-coder consistency,” Accident Analysis and Prevention, vol. 42, no. 2, pp. 437–444, 2010.
[25]
J. Reason, Human Error, Cambridge University Press, New York, NY, USA, 1990.
[26]
F. Bird, Management Guide to Loss Control, Institute Press, Atlanta, Ga, USA, 1974.
[27]
W. Johnson, “Classifying pilot human error causes in A-10 class A mishaps,” Graduate Research Project, Embrey-Riddle Aeronautical University, 1997.
[28]
Y. AlWardi, Applying human factors analysis and classification system (HFACS) to aviation accidents in the Royal Air Force of Oman (RAFO): a cross cultural comparison, M.S. thesis, Cranfield University, Bedfordshire, UK, 2006.
[29]
D. Walker, “Applying the human factors analysis and classification system (HFACS) to incidents in the UK construction industry,” Graduate Research Project, Cranfield University, 2007.
[30]
D. C. Krulak, “Human factors in maintenance: impact on aircraft mishap frequency and severity,” Aviation Space and Environmental Medicine, vol. 75, no. 5, pp. 429–432, 2004.
[31]
J. M. Patterson and S. A. Shappell, “Operator error and system deficiencies: analysis of 508 mining incidents and accidents from Queensland, Australia using HFACS,” Accident Analysis and Prevention, vol. 42, no. 4, pp. 1379–1385, 2010.
[32]
S. Reinach and A. Viale, “Application of a human error framework to conduct train accident/incident investigations,” Accident Analysis and Prevention, vol. 38, no. 2, pp. 396–406, 2006.
[33]
M. T. Baysari, C. Caponecchia, A. S. McIntosh, and J. R. Wilson, “Classification of errors contributing to rail incidents and accidents: a comparison of two human error identification techniques,” Safety Science, vol. 47, no. 7, pp. 948–957, 2009.
[34]
M. Celik and S. Cebi, “Analytical HFACS for investigating human errors in shipping accidents,” Accident Analysis and Prevention, vol. 41, no. 1, pp. 66–75, 2009.
[35]
L. R. James, R. G. Demaree, and G. Wolf, “Estimating within-group interrater reliability with and without response bias,” Journal of Applied Psychology, vol. 69, no. 1, pp. 85–98, 1984.
[36]
A. Isaac, M. Lyons, T. Bove, and D. Van Damme, “Validation of the human error in ATM (HERA-JANUS) technique,” EUROCONTROL, HRA/HSP-002-REP-04, 2003.
[37]
A. J. Ross, B. Wallace, and J. B. Davies, “Technical note: measurement issues in taxonomic reliability,” Safety Science, vol. 42, no. 8, pp. 771–778, 2004.
[38]
J. Beaubien and D. Baker, “A review of selected aviation human factors taxonomies, accident/incident reporting systems and data reporting tools,” International Journal of Applied Aviation Studies, vol. 2, no. 2, pp. 11–36, 2002.
[39]
M. Dambier and J. Hinkelbein, “Analysis of 2004 German general aviation aircraft accidents according to the HFACS model,” Air Medical Journal, vol. 25, no. 6, pp. 265–269, 2006.
