MODEL-BASED FEATURE SELECTION AND CLUSTERING OF RNA-SEQ DATA FOR UNSUPERVISED SUBTYPE DISCOVERY

• 成果类型: Article
• 署名作者: Lim, David K.; Rashid, Naim U.; Ibrahim, Joseph G.
• 署名单位: University of North Carolina; University of North Carolina Chapel Hill; University of North Carolina School of Medicine
• 刊物名称: ANNALS OF APPLIED STATISTICS
• ISSN/ISSBN: 1932-6157
• DOI: 10.1214/20-AOAS1407
• 发表日期: 2021
• 页码: 481-508
• 关键词: maximum-likelihood-estimation; variable selection; molecular portraits; em algorithm; cancer; CLASSIFICATION; regression; quantification; regularization; identification

摘要

Clustering is a form of unsupervised learning that aims to uncover latent groups within data based on similarity across a set of features. A common application of this in biomedical research is in delineating novel cancer subtypes from patient gene expression data, given a set of informative genes. However, it is typically unknown a priori what genes may be informative in discriminating between clusters and what the optimal number of clusters are. Few methods exist for performing unsupervised clustering of RNA-seq samples, and none currently adjust for between-sample global normalization factors, select cluster-discriminatory genes or account for potential confounding variables during clustering. To address these issues, we propose the feature selection and clustering of RNA-seq (FSCseq): a model-based clustering algorithm that utilizes a finite mixture of regression (FMR) model and the quadratic penalty method with a smoothly clipped absolute deviation (SCAD) penalty. The maximization is done by a penalized Classification EM algorithm, allowing us to include normalization factors and confounders in our modeling framework. Given the fitted model, our framework allows for subtype prediction in new patients via posterior probabilities of cluster membership, even in the presence of batch effects. Based on simulations and real data analysis, we show the advantages of our method relative to competing approaches.