AI-AUGMENTED CONTENT VALIDATION IN BEHAVIORAL RESEARCH: DEVELOPMENT AND EVALUATION OF THE RATER SYSTEM1

• 成果类型: Article
• 署名作者: Pillet, Jean-Charles; Larsen, Kai R.; Dobolyi, David; Queiroz, Magno; Handler, Abram; Arnulf, Jan Ketil; Sharma, Rajeev
• 署名单位: University of Colorado System; University of Colorado Boulder; State University System of Florida; Florida Atlantic University; Norwegian Defence University College; BI Norwegian Business School; Deakin University
• 刊物名称: MIS QUARTERLY
• ISSN/ISSBN: 0276-7783
• DOI: 10.25300/MISQ/2025/18946
• 发表日期: 2026-03
• 页码: 59-86
• 关键词: content validity; psychometrics; SCALE DEVELOPMENT; large language models (LLMs); Machine Learning; design science research; behavioral research; research rigor; PSYCHOLOGICAL-ASSESSMENT; content validity; information-technology; quantitative approach; construct measurement; SCIENCE RESEARCH; recommendations; DEFINITIONS; personality; acceptance

摘要

Content validation is an essential aspect of the scale development process that ensures that measurement instruments capture their intended constructs. However, researchers rarely undertake this core step in behavioral research because it requires costly data collection and specialized expertise. We present RATER (replicable approach to expert ratings), a free web-based system (www.contval.org) that can help the broader research community (scientists, reviewers, students) gain quick and reliable insights into the content validity of measurement instruments. Guided by psychometric measurement theory, RATER evaluates whether a scale's items correspond to their intended construct, remain distinct from other constructs, and adequately represent all aspects of the construct's content domain. The system employs two unique artificial intelligence models, RATERC and RATERD, which leverage psychometric scales from 2,443 journal articles spanning eight disciplines and two state-of-the-art large language model architectures (i.e., BERT and GPT). A set of six complementary studies confirms the RATER system's accuracy, reliability, and usefulness. We find that RATER can augment the scale development and validation process, increasing the validity of findings in behavioral research.