Heterogeneity in effect size estimates

• 成果类型: Article
• 署名作者: Holzmeister, Felix; Johannesson, Magnus; Boehm, Robert; Dreber, Anna; Huber, Juergen; Kirchler, Michael
• 署名单位: University of Innsbruck; Stockholm School of Economics; University of Vienna; University of Copenhagen; University of Copenhagen; University of Innsbruck
• 刊物名称: PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA
• ISSN/ISSBN: 0027-13682
• DOI: 10.1073/pnas.2403490121
• 发表日期: 2024-08-06
• 关键词: registered replication report; conducting metaanalyses; publication bias; tests; reproducibility; replicability; transparent; ECONOMICS; analysts; MODEL

摘要

A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta- analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies.