Conformal Prediction After Data-Dependent Model Selection

• 成果类型: Article; Early Access
• 署名作者: Liang, Ruiting; Zhu, Wanrong; Barber, Rina Foygel
• 署名单位: University of Chicago; University of California System; University of California Irvine; University of Chicago
• 刊物名称: JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION
• ISSN/ISSBN: 0162-1459; 1537-274X
• DOI: 10.1080/01621459.2026.2663588
• 发表日期: 2026-06-22
• 关键词: Distribution-free inference; Nonparametric Statistics; Selective Inference

摘要

Given a family of pretrained models and a hold-out set, how can we construct a valid conformal prediction set while selecting a model that minimizes the width of the set? If we use the same hold-out dataset both to select a model (the model that yields the smallest conformal prediction sets) and then to construct a conformal prediction set based on that selected model, we suffer a loss of coverage due to selection bias. Alternatively, we could further split the data to perform selection and calibration separately, but this comes at a steep cost if the size of the dataset is limited. In this article, we address the challenge of constructing a valid prediction set after data-dependent model selection-commonly, selecting the model that minimizes the width of the resulting prediction sets. Our novel methods can be implemented efficiently and admit finite-sample validity guarantees without invoking additional sample-splitting. We show that our methods yield prediction sets with asymptotically optimal width under certain notions of regularity for the model class. The improvement in the width of the prediction sets constructed by our methods is further demonstrated through applications to synthetic datasets in various settings and a real data example. for this article are available online, including a standardized description of the materials available for reproducing the work.