Efficient Human-in-the-Loop Active Learning: A Novel Framework for Data Labeling in AI Systems

• 成果类型: Article
• 署名作者: Huang, Yiran; Yang, Jian-Feng; Fu, Haoda
• 署名单位: Nankai University; University of North Carolina; University of North Carolina Chapel Hill; University of North Carolina School of Medicine
• 刊物名称: JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION
• ISSN/ISSBN: 0162-1459; 1537-274X
• DOI: 10.1080/01621459.2026.2656455
• 发表日期: 2026-04-03
• 页码: 1013-1024
• 关键词: Data labeling; dynamic prediction; Exploration and exploitation; Flexible query design; Full and partial information; Information gain; parallel

摘要

Modern AI systems rely heavily on labeled data, yet labeling is often expensive and labor-intensive, especially when requiring special skills such as reading radiology images by physicians. To most efficiently use experts' time for data labeling, one promising approach is human-in-the-loop active learning. However, traditional active learning methods are limited to single-label queries and fail to leverage more flexible query types in many real-world settings. In this work, we propose a novel active learning framework with significant potential for application in modern AI systems. This framework incorporates different query schemes and jointly determines which question to ask and which data points to query. We also introduce a method to integrate full and partial information obtained from these diverse queries. In addition, we develop an innovative model-agnostic exploration and exploitation framework to filter out redundant samples. Experiments on five datasets, including two real-world image datasets, demonstrate that the proposed framework substantially outperforms all other methods. These results highlight the framework's promise for applications across a range of scientific domains. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.