Honest calibration assessment for binary outcome predictions

• 成果类型: Article
• 署名作者: Dimitriadis, Timo; Dumbgen, Lutz; Henzi, Alexander; Puke, Marius; Ziegel, Johanna
• 署名单位: Ruprecht Karls University Heidelberg; University of Bern; Swiss Federal Institutes of Technology Domain; ETH Zurich; University Hohenheim
• 刊物名称: BIOMETRIKA
• ISSN/ISSBN: 0006-3444
• DOI: 10.1093/biomet/asac068
• 发表日期: 2023
• 页码: 663680
• 关键词: confidence; bootstrap

摘要

Probability predictions from binary regressions or machine learning methods ought to be calibrated: if an event is predicted to occur with probability x, it should materialize with approximately that frequency, which means that the so-called calibration curvep(middot) should equal the identity, i.e., p(x) = x for all x in the unit interval. We propose honest calibration assessment based on novel confidence bands for the calibration curve, which are valid subject to only the natural assumption of isotonicity. Besides testing the classical goodness-of-fit null hypothesis of perfect calibration, our bands facilitate inverted goodness-of-fit tests whose rejection allows for the sought-after conclusion of a sufficiently well-specified model. We show that our bands have a finite-sample coverage guarantee, are narrower than those of existing approaches, and adapt to the local smoothness of the calibration curve p and the local variance of the binary observations. In an application to modelling predictions of an infant having low birth weight, the bounds give informative insights into model calibration.