Document Type

Conference Paper

Publication Date

2018

Publication Title

Proceedings of the American Society for Engineering Management 2018 International Annual Conference

Pages

1-13

Conference Name

39th American Society for Engineering Management International Annual Conference, 17-20 October 2018, Coeur d'Alene, Idaho, USA

Abstract

Inter-rater reliability can be regarded as the degree of agreement among raters on a given item or a circumstance. Multiple approaches have been taken to estimate and improve inter-rater reliability of the United States Department of Defense Human Factors Analysis and Classification System used by trained accident investigators. In this study, three trained instructor pilots used the DoD-HFACS to classify 347 U.S. Air Force Accident Investigation Board (AIB) Class-A reports between the years of 2000 and 2013. The overall method consisted of four steps: (1) train on HFACS definitions, (2) verify rating reliability, (3) rate HFACS reports, and (4) random sample to validate ratings reliability. Attribute agreement analysis was used as the method to assess inter-rater reliability. In the final training verification round, within appraiser agreement ranged 85.28% to 93.25%, each appraiser versus the standard ranged 77.91% to 82.82%, between appraisers 72.39%, and all appraisers versus the standard was 67.48%. Corresponding agreement for the random sample of HFACS rated summaries were within appraiser 78.89% to 92.78% and between appraisers 53.33%, which is consistent with prior studies. This pilot study indicates that the train-verify-rate-validate attribute agreement analysis approach has the potential to aid in improving HFACS ratings reliability and contributing to accurately capturing human factors contributions to aircraft mishaps. Additional full-scale studies will be required to verify and fully develop the proposed methodology.

Comments

Included with the kind permission of the publisher.

Copyright, American Society for Engineering Management, 2018

Original Publication Citation

Cotter, T. S., & Yesilbas, V. (2018). An attribute agreement analysis method for HFACS inter-rater reliability assessment. In E-H. Ng, B. Nepal, E. Schott, & H. Keathley (Eds.), Proceedings of the American Society for Engineering Management 2018 International Annual Conference (pp. 1-13). American Society for Engineering Management (ASEM).

Download

Share

COinS