The A/B testing problem with Gaussian priors

• 成果类型: Article
• 署名作者: Azevedo, Eduardo M.; Mao, David; Olea, Jose Luis Montiel; Velez, Amilcar
• 署名单位: University of Pennsylvania; University of Pennsylvania; Cornell University; Northwestern University
• 刊物名称: JOURNAL OF ECONOMIC THEORY
• ISSN/ISSBN: 0022-0531
• DOI: 10.1016/j.jet.2023.105646
• 发表日期: 2023
• 关键词: Statistical decision theory; Optimal learning; Experiment design; A; B testing

摘要

A risk-neutral firm can perform a randomized experiment (A/B test) to learn about the effects of im-plementing an idea of unknown quality. The firm's goal is to decide the experiment's sample size and whether or not the idea should be implemented after observing the experiment's outcome. We show that when the distribution for idea quality is Gaussian and there are linear costs of experimentation, there are exact formulae for the firm's optimal implementation decisions, the value of obtaining more data, and op-timal experiment sizes. Our formulae-which assume that companies use randomized experiments to help them maximize expected profits-provide a simple alternative to i) the standard rules-of-thumb of power calculations for determining the sample size of an experiment, and also to ii) ad hoc thresholds based on statistical significance to interpret the outcome of an experiment. (c) 2023 Published by Elsevier Inc.