Leveraging Expert Consistency to Improve Algorithmic Decision Support

• 成果类型: Article; Early Access
• 署名作者: De-Arteaga, Maria; Jeanselme, Vincent; Dubrawski, Artur; Chouldechova, Alexandra
• 署名单位: University of Texas System; University of Texas Austin; University of Cambridge; MRC Biostatistics Unit; Carnegie Mellon University; Carnegie Mellon University
• 刊物名称: MANAGEMENT SCIENCE
• ISSN/ISSBN: 0025-1909
• DOI: 10.1287/mnsc.2022.01576
• 发表日期: 2025
• 关键词: predictive algorithms; Decision Support; Machine Learning; design science; expert consistency; construct gap

摘要

Machine learning (ML) is increasingly being used to support high-stakes decisions. However, there is frequently a construct gap: a gap between the construct of interest to the decision-making task and what is captured in proxies used as labels to train ML models. As a result, ML models may fail to capture important dimensions of decision criteria, hampering their utility for decision support. Thus, an essential step in the design of ML systems for decision support is selecting a target label among available proxies. In this work, we explore the use of historical expert decisions as a rich-yet also imperfect- source of information that can be combined with observed outcomes to narrow the construct gap. We argue that managers and system designers may be interested in learning from experts in instances where they exhibit consistency with each other while learning from observed outcomes otherwise. We develop a methodology to enable this goal using information that is commonly available in organizational information systems. This involves two core steps. First, we propose an influence function-based methodology to estimate expert consistency indirectly when each case in the data are assessed by a single expert. Second, we introduce a label amalgamation approach that allows ML models to simultaneously learn from expert decisions and observed outcomes. Our empirical evaluation, using simulations in a clinical setting and real-world data from the child welfare domain, indicates that the proposed approach successfully narrows the construct gap, yielding better predictive performance than learning from either observed outcomes or expert decisions alone.