A Comparison of Methods for Treatment Assignment with an Application to Playlist Generation

• 成果类型: Article
• 署名作者: Fernandez-Loria, Carlos; Provost, Foster; Anderton, Jesse; Carterette, Benjamin; Chandar, Praveen
• 署名单位: Hong Kong University of Science & Technology; New York University; Spotify
• 刊物名称: INFORMATION SYSTEMS RESEARCH
• ISSN/ISSBN: 1047-7047
• DOI: 10.1287/isre.2022.1149
• 发表日期: 2023
• 页码: 786-803
• 关键词: treatment rules

摘要

This study presents a systematic comparison of methods for individual treatment assignment, a general problem that arises in many applications and that has received significant attention fromeconomists, computer scientists, and social scientists. We group the various methods proposed in the literature into three general classes of algorithms (or metalearners): learning models to predict outcomes (the O-learner), learning models to predict causal effects (the E-learner), and learning models to predict optimal treatment assignments (the A-learner). We compare themetalearners in terms of (1) their level of generality and (2) the objective function they use to learnmodels fromdata; we then discuss the implications that these characteristics have formodeling and decisionmaking. Notably, we demonstrate analytically and empirically that optimizing for the prediction of outcomes or causal effects is not the same as optimizing for treatment assignments, suggesting that, in general, the A-learner should lead to better treatment assignments than the othermetalearners. We demonstrate the practical implications of our findings in the context of choosing, for each user, the best algorithm for playlist generation in order to optimize engagement. This is the first comparison of the three different metalearners on a real-world application at scale (based on more than half a billion individual treatment assignments). In addition to supporting our analytical findings, the results show how large A/B tests can provide substantial value for learning treatment-assignment policies, rather than simply for choosing the variant that performs best on average.