Comparing Sequential Forecasters

• 成果类型: Article
• 署名作者: Choe, Yo Joong; Ramdas, Aaditya
• 署名单位: University of Chicago; Carnegie Mellon University
• 刊物名称: OPERATIONS RESEARCH
• ISSN/ISSBN: 0030-364X
• DOI: 10.1287/opre.2021.0792
• 发表日期: 2024
• 页码: 1368-1387
• 关键词: boundary crossing probabilities; Proper scoring rules; time-uniform; calibration; expectiles; quantiles; tests; sums

摘要

Consider two forecasters, each making a single prediction for a sequence of events over time. We ask a relatively basic question: how might we compare these forecasters, either online or post hoc, avoiding unverifiable assumptions on how the forecasts and outcomes were generated? In this paper, we present a rigorous answer to this question by designing novel sequential inference procedures for estimating the time-varying difference in forecast scores. To do this, we employ confidence sequences (CS), which are sequences of confidence intervals that can be continuously monitored and are valid at arbitrary data dependent stopping times (anytime-valid). The widths of our CSs are adaptive to the underlying variance of the score differences. Underlying their construction is a game theoretic statistical framework in which we further identify e-processes and p-processes for sequentially testing a weak null hypothesis-whether one forecaster outperforms another on average (rather than always). Our methods do not make distributional assumptions on the forecasts or outcomes; our main theorems apply to any bounded scores, and we later provide alternative methods for unbounded scores. We empirically validate our approaches by comparing real-world baseball and weather forecasters.