CAN QUASI-EXPERIMENTAL EVALUATIONS THAT RELY ON STATE LONGITUDINAL DATA SYSTEMS REPLICATE EXPERIMENTAL RESULTS?

• 成果类型: Article
• 署名作者: Unlu, Fatih; Lauen, Douglas Lee; Fuller, Sarah Crittenden; Berglund, Tiffany; Estrera, Elc
• 署名单位: RAND Corporation; University of North Carolina; University of North Carolina Chapel Hill; University of North Carolina; University of North Carolina Chapel Hill; University of North Carolina; University of North Carolina Chapel Hill
• 刊物名称: JOURNAL OF POLICY ANALYSIS AND MANAGEMENT
• ISSN/ISSBN: 1520-6733
• DOI: 10.1002/pam.22295
• 发表日期: 2021
• 页码: 572-613
• 关键词: crime-prevention; cctv; SURVEILLANCE; Cameras

摘要

Do quasi-experimental (QE) studies conducted with baseline covariates that are typically available in the longitudinal administrative state databases yield unbiased effectestimates? This paper conducts a within-study comparison (WSC) study that comparesexperimental impacts of early college high school (ECHS) attendance with QE impactsdrawn from the state and locales. We find that (1) QE models for outcomes with natural(matching) pretests replicated the randomized benchmarks quite well; (2) the replica-tion bias is not sensitive to type of propensity score model or method; and (3) imposinglocational restrictions (i.e., local matching) on the comparison students-specificallychoosing them from among non-treatment students who came from the same feedermiddle schools as the treatment students-does not decrease the QE bias; on the con-trary, it performed worse than the models that did not impose this restriction for mostoutcomes. The first two findings are generally consistent with other education WSCswhile the third one is not, suggesting that in cases where selection may be driven byindividual-level factors, such as this one, local matching may yield biased treatment effect estimates by greatly reducing the pool of potential comparison units and distortingbalance on unobservable confounders while prioritizing balance on observable factors. (c) 2021 by the Association for Public Policy Analysis and Management.