Improved Small-Sample Estimation of Nonlinear Cross-Validated Prediction Metrics

• 成果类型: Article
• 署名作者: Benkeser, David; Petersen, Maya; van der Laan, Mark J.
• 署名单位: Emory University; University of California System; University of California Berkeley; University of California System; University of California Berkeley
• 刊物名称: JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION
• ISSN/ISSBN: 0162-1459
• DOI: 10.1080/01621459.2019.1668794
• 发表日期: 2020
• 页码: 1917-1932
• 关键词: models; CLASSIFICATION

摘要

When predicting an outcome is the scientific goal, one must decide on a metric by which to evaluate the quality of predictions. We consider the problem of measuring the performance of a prediction algorithm with the same data that were used to train the algorithm. Typical approaches involve bootstrapping or cross-validation. However, we demonstrate that bootstrap-based approaches often fail and standard cross-validation estimators may perform poorly. We provide a general study of cross-validation-based estimators that highlights the source of this poor performance, and propose an alternative framework for estimation using techniques from the efficiency theory literature. We provide a theorem establishing the weak convergence of our estimators. The general theorem is applied in detail to two specific examples and we discuss possible extensions to other parameters of interest. For the two explicit examples that we consider, our estimators demonstrate remarkable finite-sample improvements over standard approaches. for this article are available online.