Multiple Model Evaluation Absent the Gold Standard Through Model Combination

• 成果类型: Article
• 署名作者: Iversen, Edwin S.; Parmigiani, Giovanni; Chen, Sining
• 署名单位: Duke University; Johns Hopkins University; Johns Hopkins University; Johns Hopkins University; Johns Hopkins University; Johns Hopkins University
• 刊物名称: JOURNAL OF THE AMERICAN STATISTICAL ASSOCIATION
• ISSN/ISSBN: 0162-1459
• DOI: 10.1198/016214507000001012
• 发表日期: 2008
• 页码: 897-909
• 关键词: ovarian-cancer; brca2 mutations; carrier probabilities; germline mutations; predicting brca1; genetic-analysis; family-history; breast-cancer; ERROR RATES; RISK

摘要

We describe a method for evaluating an ensemble of predictive models given a sample of observations comprising the model predictions and the outcome event measured with error. Our formulation allows us to simultaneously estimate measurement error parameters, true outcome-the gold standard-and a relative weighting of the predictive scores. We describe conditions necessary to estimate the gold standard and to calibrate these estimates and detail how our approach is related to, but distinct from, standard model combination techniques. We apply our approach to data from a study to evaluate a collection of BRCA1/BRCA2 gene mutation prediction scores. In this example, genotype is measured with error by one or more genetic assays. We estimate true genotype for each individual in the data set, operating characteristics of the commonly used genotyping procedures, and a relative weighting of the scores. Finally, we compare the scores against the gold standard genotype and find that Mendelian scores, on average, the more refined and better calibrated of those considered and that the comaprison is sensitive to measurement error in the gold standard.