Reducing Interference Bias in Online Marketplace Experiments Using Cluster Randomization: Evidence from a Pricing Meta-experiment on Airbnb

• 成果类型: Article
• 署名作者: Holtz, David; Lobel, Felipe; Lobel, Ruben; Liskovich, Inessa; Aral, Sinan
• 署名单位: University of California System; University of California Berkeley; Massachusetts Institute of Technology (MIT); University of California System; University of California Berkeley; Airbnb; Massachusetts Institute of Technology (MIT)
• 刊物名称: MANAGEMENT SCIENCE
• ISSN/ISSBN: 0025-1909
• DOI: 10.1287/mnsc.2020.01157
• 发表日期: 2025
• 关键词: design of experiments; electronic markets and auctions; interference; cluster randomization; Airbnb

摘要

Online marketplace designers frequently run randomized experiments to measure the impact of proposed product changes. However, given that marketplaces are inherently connected, total average treatment effect (TATE) estimates obtained through individual -level randomized experiments may be biased because of violations of the stable unit treatment value assumption, a phenomenon we refer to as interference bias. Cluster randomization (i.e., the practice of randomizing treatment assignment at the level of clusters of similar individuals) is an established experiment design technique for countering interference bias in social networks, but it is unclear ex ante if it will be effective in marketplace settings. In this paper, we use a meta -experiment or experiment over experiments conducted on Airbnb to both provide empirical evidence of interference bias in online marketplace settings and assess the viability of cluster randomization as a tool for reducing interference bias in marketplace TATE estimates. Results from our metaexperiment indicate that at least 20% of the TATE estimate produced by an individuallevel randomized evaluation of the platform fee increase we study is attributable to interference bias and eliminated through the use of cluster randomization. We also find suggestive, nonstatistically significant evidence that interference bias in seller -side experiments is more severe in demand -constrained geographies and that the efficacy of cluster randomization at reducing interference bias increases with cluster quality.