A/B Testing with Fat Tails

• 成果类型: Article
• 署名作者: Azevedo, Eduardo M.; Deng, Alex; Montiel Olea, Jose Luis; Rao, Justin; Weyl, E. Glen
• 署名单位: University of Pennsylvania; Microsoft; Columbia University; Microsoft; Princeton University
• 刊物名称: JOURNAL OF POLITICAL ECONOMY
• ISSN/ISSBN: 0022-3808
• DOI: 10.1086/710607
• 发表日期: 2020
• 页码: 4614-4672
• 关键词: EMPIRICAL BAYES ESTIMATION; likelihood; decisions; CITATION; demand; POWER; LAW

摘要

We propose a new framework for optimal experimentation, which we term the A/B testing problem. Our model departs from the existing literature by allowing for fat tails. Our key insight is that the optimal strategy depends on whether most gains accrue from typical innovations or from rare, unpredictable large successes. If the tails of the unobserved distribution of innovation quality are not too fat, the standard approach of using a few high-powered big experiments is optimal. However, if the distribution is very fat tailed, a lean strategy of trying more ideas, each with possibly smaller sample sizes, is preferred. Our theoretical results, along with an empirical analysis of Microsoft Bing's EXP platform, suggest that simple changes to business practices could increase innovation productivity.