PHASE TRANSITIONS FOR HIGH DIMENSIONAL CLUSTERING AND RELATED PROBLEMS

• 成果类型: Article
• 署名作者: Jin, Jiashun; Ke, Zheng Tracy; Wang, Wanjie
• 署名单位: Carnegie Mellon University; University of Chicago; University of Pennsylvania
• 刊物名称: ANNALS OF STATISTICS
• ISSN/ISSBN: 0090-5364
• DOI: 10.1214/16-AOS1522
• 发表日期: 2017
• 页码: 2151-2189
• 关键词: higher criticism; SPARSE; CLASSIFICATION; inference; selection; features; rare; PCA

摘要

Consider a two-class clustering problem where we observe X-i = l(i)mu + Zi, Zi((i,i,d) under tilde) N(0, I-p), 1 <= i <= n. The feature vector mu is an element of R-p is unknown but is presumably sparse. The class labels l(i) is an element of {-1, 1} are also unknown and the main interest is to estimate them. We are interested in the statistical limits. In the two-dimensional phase space calibrating the rarity and strengths of useful features, we find the precise demarcation for the Region of Impossibility and Region of Possibility. In the former, useful features are too rare/ weak for successful clustering. In the latter, useful features are strong enough to allow successful clustering. The results are extended to the case of colored noise using Le Cam's idea on comparison of experiments. We also extend the study on statistical limits for clustering to that for signal recovery and that for global testing. We compare the statistical limits for three problems and expose some interesting insight. We propose classical PCA and Important Features PCA (IF-PCA) for clustering. For a threshold t > 0, IF-PCA clusters by applying classical PCA to all columns of X with an L-2-norm larger than t. We also propose two aggregation methods. For any parameter in the Region of Possibility, some of these methods yield successful clustering. We discover a phase transition for IF-PCA. For any threshold t > 0, let xi ((t)) be the first left singular vector of the post-selection data matrix. The phase space partitions into two different regions. In one region, there is a t such that cos(xi((t)), l) -> 1 and IF-PCA yields successful clustering. In the other, cos(xi((t)), l) <= c(0) < 1 for all t > 0. Our results require delicate analysis, especially on post-selection random matrix theory and on lower bound arguments.