Who r u?: On the (In) accuracy of Incumbent-Based Estimates of Range Restriction in Criterion-Related and Differential Validity Research

• 成果类型: Article
• 署名作者: Roth, Philip L.; Le, Huy; Oh, In-Sue; Van Iddekinge, Chad H.; Robbins, Steven B.
• 署名单位: Clemson University; University of Texas System; University of Texas at San Antonio; Pennsylvania Commonwealth System of Higher Education (PCSHE); Temple University; State University System of Florida; Florida State University; Educational Testing Service (ETS)
• 刊物名称: JOURNAL OF APPLIED PSYCHOLOGY
• ISSN/ISSBN: 0021-9010
• DOI: 10.1037/apl0000193
• 发表日期: 2017
• 页码: 802-828
• 关键词: cognitive ability tests; differential validity; Meta-analysis; personnel selection; RANGE RESTRICTION

摘要

Correcting validity estimates for selection procedures for range restriction typically involves comparing variance in predictor scores between all job applicants and applicants who were selected. However, some research on criterion-related and differential validity of cognitive ability tests has relied on range restriction corrections based on data from job incumbents. Unfortunately, there remains ambiguity concerning the accuracy of this incumbent-based approach vis-a-vis the applicant-based approach. To address this issue, we conducted several Monte Carlo simulations, as well as an analysis of college admissions data. Our first simulation study showed that incumbent-based range restriction corrections result in downwardly biased estimates of criterion-related validity, whereas applicant-based corrections were quite accurate. Our second set of simulations showed that incumbent-based range restriction corrections can produce evidence of differential validity when there is no differential validity in the population. In contrast, applicant-based corrections tended to accurately estimate population parameters and showed little, if any, evidence of differential validity when there is no differential validity in the population. Analysis of data for the ACT as a predictor of academic performance revealed similar patterns of bias for incumbent-based corrections in an academic setting. Overall, the present findings raise serious concerns regarding the use of incumbent-based range restriction corrections in lieu of applicant-based corrections. They also cast doubt on recent evidence for differential validity of predictors of job performance.