Do Evidence-Based Policy Clearinghouses Provide Good Advice for Local Policymakers?

• 成果类型: Article
• 署名作者: Orr, Larry L.
• 署名单位: Johns Hopkins University; Johns Hopkins Bloomberg School of Public Health
• 刊物名称: JOURNAL OF POLICY ANALYSIS AND MANAGEMENT
• ISSN/ISSBN: 1520-6688
• DOI: 10.1002/pam.70077
• 发表日期: 2025-12-29
• 页码: e70077
• 关键词: TREATMENT CHOICE; reproducibility; Heterogeneity; selection; validity; BIAS

摘要

Policymakers are often urged to rely on evidence-based policy (EBP)-adopting only interventions proven effective (i.e., positive and statistically significant in multisite impact evaluations). EBP clearinghouses chronicle and rate tests of social policy interventions. But EBP clearinghouse standards are based almost entirely on internal validity. They largely ignore whether research findings from multisite trials apply to individual localities, where much of social policy is formulated. We develop a Bayesian model of the probability that the EBP rule is sound advice to local policymakers. The model allows a direct test of the probability of a correct policy decision under the EBP rule, its positive predictive value (PPV), and its negative predictive value (NPV)-the probabilities that an intervention deemed effective by that rule will in fact be effective in a particular site (PPV), and that an intervention deemed ineffective will not be effective in a particular site (NPV), given the true impact of the intervention. These intuitive, easily calculated probabilities are major contributions of this paper. In our illustrative analysis of six multisite randomized trials, we find that under the EBP clearinghouse rule the probability of a correct policy decision, PPV, and NPV are all unacceptably low unless the cross-site impact heterogeneity is quite low.