SHARP DETECTION IN PCA UNDER CORRELATIONS: ALL EIGENVALUES MATTER

• 成果类型: Article
• 署名作者: Dobriban, Edgar
• 署名单位: Stanford University
• 刊物名称: ANNALS OF STATISTICS
• ISSN/ISSBN: 0090-5364
• DOI: 10.1214/16-AOS1514
• 发表日期: 2017
• 页码: 1810-1833
• 关键词: sample covariance matrices; linear spectral statistics; asymptotics; clt

摘要

Principal component analysis (PCA) is a widely used method for dimension reduction. In high-dimensional data, the signal eigenvalues corresponding to weak principal components (PCs) do not necessarily separate from the bulk of the noise eigenvalues. Therefore, popular tests based on the largest eigenvalue have little power to detect weak PCs. In the special case of the spiked model, certain tests asymptotically equivalent to linear spectral statistics (LSS)-averaging effects over all eigenvalues-were recently shown to achieve some power. We consider a local alternatives model for the spectrum of covariance matrices that allows a general correlation structure. We develop new tests to detect PCs in this model. While the top eigenvalue contains little information, due to the strong correlations between the eigenvalues we can detect weak PCs by averaging over all eigenvalues using LSS. We show that it is possible to find the optimal LSS, by solving a certain integral equation. To solve this equation, we develop efficient algorithms that build on our recent method for computing the limit empirical spectrum [Dobriban (2015)]. The solvability of this equation also presents a new perspective on phase transitions in spiked models.