INTERNAL AND EXTERNAL VALIDITY OF THE COMPARATIVE INTERRUPTED TIME-SERIES DESIGN: A META-ANALYSIS

• 成果类型: Article
• 署名作者: Coopersmith, Jared; Cook, Thomas D.; Zurovac, Jelena; Chaplin, Duncan; Forrow, Lauren, V
• 署名单位: Mathematica; Northwestern University; George Washington University; Mathematica
• 刊物名称: JOURNAL OF POLICY ANALYSIS AND MANAGEMENT
• ISSN/ISSBN: 1520-6983
• DOI: 10.1002/pam.22361
• 发表日期: 2022
• 页码: 252-+
• 关键词: randomized experiment; CONSEQUENCES; performance; PROGRAMS; earnings

摘要

This paper meta-analyzes 12 heterogeneous studies that examine bias in the comparative interrupted time-series design (CITS) that is often used to evaluate the effects of social policy interventions. To measure bias, each CITS impact estimate was differenced from the estimate derived from a theoretically unbiased causal benchmark study that tested the same hypothesis with the same treatment group, outcome data, and estimand. In 10 studies, the benchmark was a randomized experiment and in the other two it was a regression-discontinuity study. Analyses revealed the average standardized CITS bias to be between -0.01 and 0.042 standard deviations; and all but one bias estimate from individual studies fell within 0.10 standard deviations of its benchmark, indicating that the near zero mean bias did not result from averaging many large single study differences. The low mean and generally tight distribution of individual bias estimates suggest that CITS studies are worth recommending for future causal hypothesis tests because: (1) over the studies examined, they generally resulted in high internal validity; and (2) they also promise high external validity because the empirical tests we synthesized occurred across a wide variety of settings, times, interventions, and outcomes.