A RANDOM MATRIX APPROACH TO NEURAL NETWORKS

• 成果类型: Article
• 署名作者: Louart, Cosme; Liao, Zhenyu; Couillet, Romain
• 署名单位: Universite Paris Saclay
• 刊物名称: ANNALS OF APPLIED PROBABILITY
• ISSN/ISSBN: 1050-5164
• DOI: 10.1214/17-AAP1328
• 发表日期: 2018
• 页码: 1190-1248
• 关键词: limiting spectral distribution; extreme learning-machine; eigenvalues; MODEL; estimator; systems

摘要

This article studies the Gram random matrix model G = 1/T Sigma E-T E = sigma (WX), classically found in the analysis of random feature maps and random neural networks, where X = [x(1),, x(T)] epsilon R-PXT is a (data) matrix of bounded norm, W epsilon R-nxp) is a matrix of independent zero-mean unit variance entries and o : R -> R is a Lipschitz continuous (activation) function-sigma (WX) being understood entry-wise. By means of a key concentration of measure lemma arising from nonasymptotic random matrix arguments, we prove that, as n, p, T grow large at the same rate, the resolvent Q = (G + gamma I-T)(-1), for gamma > 0, has a similar behavior as that met in sample covariance matrix models, involving notably the moment.phi = T/n, E[G], which provides in passing a deterministic equivalent for the empirical spectral measure of G. Application-wise, this result enables the estimation of the asymptotic performance of single-layer random neural networks. This in turn provides practical insights into the underlying mechanisms into play in random neural networks, entailing several unexpected consequences, as well as a fast practical means to tune the network hyperparameters.