Illusory generalizability of clinical prediction models

• 成果类型: Article
• 署名作者: Chekroud, Adam M.; Hawrilenko, Matt; Loho, Hieronimus; Bondar, Julia; Gueorguieva, Ralitza; Hasan, Alkomiet; Kambeitz, Joseph; Corlett, Philip R.; Koutsouleris, Nikolaos; Krumholz, Harlan M.; Krystal, John H.; Paulus, Martin
• 署名单位: Yale University; Yale University; University of Augsburg; University of Cologne; University of Cologne; University of Munich; Yale University; Laureate Institute for Brain Research, Inc.
• 刊物名称: SCIENCE
• ISSN/ISSBN: 0036-9625
• DOI: 10.1126/science.adg8538
• 发表日期: 2024-01-12
• 页码: 164-167
• 关键词: regularization; schizophrenia; selection; scale

摘要

It is widely hoped that statistical models can improve decision-making related to medical treatments. Because of the cost and scarcity of medical outcomes data, this hope is typically based on investigators observing a model's success in one or two datasets or clinical contexts. We scrutinized this optimism by examining how well a machine learning model performed across several independent clinical trials of antipsychotic medication for schizophrenia. Models predicted patient outcomes with high accuracy within the trial in which the model was developed but performed no better than chance when applied out-of-sample. Pooling data across trials to predict outcomes in the trial left out did not improve predictions. These results suggest that models predicting treatment outcomes in schizophrenia are highly context-dependent and may have limited generalizability.