Statistical Significance Criteria for the rWG and Average Deviation Interrater Agreement Indices

• 成果类型: Article
• 署名作者: Smith-Crowe, Kristin; Burke, Michael J.; Cohen, Ayala; Doveh, Etti
• 署名单位: Utah System of Higher Education; University of Utah; Tulane University; Technion Israel Institute of Technology
• 刊物名称: JOURNAL OF APPLIED PSYCHOLOGY
• ISSN/ISSBN: 0021-9010
• DOI: 10.1037/a0034556
• 发表日期: 2014
• 页码: 239-261
• 关键词: interrater agreement; MULTILEVEL RESEARCH; r(WG); AD; data aggregation

摘要

Despite the widespread use of interrater agreement statistics for multilevel modeling and other types of research, the existing guidelines for inferring the statistical significance of interrater agreement are quite limited. They are largely relevant only under conditions that numerous researchers have argued rarely exist. Here we address this problem by generating guidelines for inferring statistical significance under a number of conditions via a computer simulation. As a set, these guidelines cover many of the conditions researchers commonly face. We discuss how researchers can use the guidelines presented to more reasonably infer the statistical significance of interrater agreement relative to using the limited guidelines available in the extant literature.