When Are Item Nuances Useful for Prediction in Organizations? Comparing the Validity of Item-Level, Scale-Level, and Ensemble Machine Learning Models

• 成果类型: Article; Early Access
• 署名作者: Tang, Chen; Hickman, Louis; Song, Q. Chelsea; Alexander III, Leo
• 署名单位: American University; Virginia Polytechnic Institute & State University; Indiana University System; IU Kelley School of Business; Indiana University Bloomington; University of Illinois System; University of Illinois Urbana-Champaign; University of Illinois System; University of Illinois Urbana-Champaign
• 刊物名称: JOURNAL OF APPLIED PSYCHOLOGY
• ISSN/ISSBN: 0021-9010
• DOI: 10.1037/apl0001339
• 发表日期: 2025
• 关键词: PERSONALITY MEASURES; JOB-PERFORMANCE; selection; PSYCHOLOGY; utility; broad; work

摘要

Recent evidence suggests that prediction models using item scores, instead of the traditionally used scale scores, can more accurately predict outcomes of interest. However, little is known about the conditions under which item- or scale-level models are more suitable for prediction in organizational practice. To address this gap, we examined several real-world data sets for the presence of valid item nuances (i.e., criterion-valid item-specific variance that is lost when aggregating to scale scores). We then designed and conducted a Monte Carlo simulation based on empirical estimates to investigate the criterion-related validity of item- and scale-level models. In the simulation, we varied (a) the distribution of nuances among items, (b) the effect size of nuances, (c) the effect size of constructs, (d) scale internal consistency, and (e) training sample size. Results suggested that item-level models are recommended when relatively few items in a scale carry nuances, the nuance effect sizes are similar or larger in magnitude to the scale effect sizes, internal consistency is high, and training sample size is large. Our review of prior studies and analyses of real-world data suggest that the conditions favoring item-level models are not uncommon in organizational data. For conditions that do not substantiate the use of either item- or scale-level models, we also examined ensemble models and found that they can be an attractive alternative when the choice is unclear. We provide R code and recommendations for examining valid nuances in one's data and developing item-level, scale-level, and ensemble predictive models.