What Do Assessment Center Ratings Reflect? Consistency and Heterogeneity in Variance Composition Across Multiple Samples

• 成果类型: Article; Early Access
• 署名作者: Ingold, Pia V.; Heimann, Anna Luca; Waller, Bettina; Breil, Simon M.; Sackett, Paul R.
• 署名单位: University of Copenhagen; University of Zurich; University of Zurich; University of Munster; University of Minnesota System; University of Minnesota Twin Cities
• 刊物名称: JOURNAL OF APPLIED PSYCHOLOGY
• ISSN/ISSBN: 0021-9010
• DOI: 10.1037/apl0001318
• 发表日期: 2025
• 关键词: assessment center; generalizability theory; reliability; Meta-analysis

摘要

The question of what assessment centers' measure has remained a controversial topic in research for decades, with a recent increase in studies that (a) use generalizability theory and (b) acknowledge the effects of aggregating postexercise dimension ratings into higher level assessment center scores. Building on these developments, we used Bayesian generalizability theory and random-effects meta-analyses to examine the variance explained by assessment center components such as assessees, exercises, dimensions, assessors, their interactions, and the interrater reliability of AC ratings in 19 different assessment center samples from various organizations (N = 4,963 assessees with 272,528 observations). This provides the first meta-analytic estimates of these effects, as well as insight into the extent to which findings from previous studies generalize to assessment center samples that differ in measurement design, industry, and purpose, and how heterogeneous these effects are across samples. Results were consistent with previous trends in the ranking of variance explained by key AC components (with assessee main effects and assessee-exercise effects being the largest variance components) and additionally emphasized the relevance of assessee-exercise-dimension effects. In addition, meta-analytic results suggested substantial heterogeneity in all reliable variance components (i.e., assessee main effect, assessee-exercise effect, assessee-dimension effect, and assessee-exercise-dimension effect) and in interrater reliability across assessment center samples. Aggregating AC ratings into higher level scores (i.e., overall AC scores, exercise-level scores, and dimension-level scores) reduced heterogeneity only slightly. Implications of the findings for a multifaceted assessment center functioning are discussed.