Regularized Aggregation of One-Off Probability Predictions

• 成果类型: Article
• 署名作者: Satopaa, Ville A.
• 署名单位: INSEAD Business School
• 刊物名称: OPERATIONS RESEARCH
• ISSN/ISSBN: 0030-364X
• DOI: 10.1287/opre.2021.2224
• 发表日期: 2022
• 页码: 3558-3580
• 关键词: SCORING RULES; distributions; forecasts; geometry; priors; CROWDS

摘要

Forecasters predicting the chances of a future event may disagree because of differing evidence or noise. To harness the collective evidence of the crowd, we propose a Bayesian aggregator that is regularized by analyzing the forecasters' disagreement and ascribing overdispersion to noise. Our aggregator requires no user intervention and can be computed efficiently even for a large number of predictions. To illustrate, we evaluate our aggregator on subjective probability predictions collected during a four-year forecasting tournament sponsored by the U.S. intelligence community. Our aggregator improves the squared error (a.k.a., the Brier score) of simple averaging by around 20% and other commonly used aggregators by 10%-25%. This advantage stems almost exclusively from improved calibration. An R package called braggR implements ourmethod and is available on CRAN.