AN EVALUATION OF STATISTICAL METHODS FOR AGGREGATE PATTERNS OF REPLICATION FAILURE

• 成果类型: Article
• 署名作者: Schauer, Jacob M.; Fitzgerald, Kaitlyn G.; Peko-Spicer, Sarah; Whalen, Mena C. R.; Zejnullahi, Rrita; Hedges, Larry, V
• 署名单位: Northwestern University; Feinberg School of Medicine; Northwestern University
• 刊物名称: ANNALS OF APPLIED STATISTICS
• ISSN/ISSBN: 1932-6157
• DOI: 10.1214/20-AOAS1387
• 发表日期: 2021
• 页码: 208-229
• 关键词: correlation-coefficient; replicability; association; persistence; tests; ratio; POWER

摘要

Several programs of research have sought to assess the replicability of scientific findings in different fields, including economics and psychology. These programs attempt to replicate several findings and use the results to say something about large-scale patterns of replicability in a field. However, little work has been done to understand the analytic methods used to do this, including what they are assessing and what their statistical properties are. This article examines several methods that have been used to study patterns of replicability in the social sciences. We describe in concrete terms how each method operationalizes the idea of replication and examine various statistical properties, including bias, precision and statistical power. We find that some analytic methods rely on an operational definition of replication that can be misleading. Other methods involve more sound definitions of replication, but most of these have limitations, such as large bias and uncertainty or low power. The findings suggest that we should use caution interpreting the results of such analyses and that work on more accurate methods may be useful to future replication research efforts.