Central Tendency and Matched Difference Approaches for Assessing Interrater Agreement

• 成果类型: Article
• 署名作者: Burke, Michael J.; Cohen, Ayala; Doveh, Etti; Smith-Crowe, Kristin
• 署名单位: Tulane University; Technion Israel Institute of Technology; Boston University
• 刊物名称: JOURNAL OF APPLIED PSYCHOLOGY
• ISSN/ISSBN: 0021-9010
• DOI: 10.1037/apl0000325
• 发表日期: 2018
• 页码: 1198-1229
• 关键词: average deviation; ICC; INTERRATER AGREEMENT; pseudo agreement; r(WG)

摘要

In Study 1 of this two-part investigation, we present a central tendency approach and procedures for assessing overall interrater agreement across multiple groups. We define parameters for mean group agreement and construct bootstrapped confidence intervals around the mean population parameters for r(WG), AD, and ICC(1). In Study 2, we extend assessments of overall interrater agreement by developing a matched difference approach and procedures for assessing real versus pseudo agreement in a sample of groups. Here, we use random group resampling and the matched difference between assessments of the respective r(WG), AD, and ICC(1) values for actual and pseudo groups, with the establishment of bootstrapped confidence intervals around such differences. In both studies, we employ simulated and real data to demonstrate the accuracy and practical utility of the new procedures for assessing agreement with respect to groups. Notably, to generate simulated data for Studies 1 and 2, we developed a new underlying model for multilevel data and procedure for data generation, and we discuss its potential utility for enhancing research in group-level studies. Moreover, we discuss, relative to current practices, how and why the new inference procedures provide information about mean interrater agreement in the population, which can improve data aggregation decisions and interpretations of findings from group-level studies.