Mechanism for feature learning in neural networks and backpropagation-free machine learning models

• 成果类型: Article
• 署名作者: Radhakrishnan, Adityanarayanan; Beaglehole, Daniel; Pandit, Parthe; Belkin, Mikhail
• 署名单位: Harvard University; Harvard University; Massachusetts Institute of Technology (MIT); Broad Institute; University of California System; University of California San Diego; Indian Institute of Technology System (IIT System); Indian Institute of Technology (IIT) - Bombay; University of California System; University of California San Diego
• 刊物名称: SCIENCE
• ISSN/ISSBN: 0036-11442
• DOI: 10.1126/science.adi5639
• 发表日期: 2024-03-29
• 页码: 1461-1467
• 关键词: regression

摘要

Understanding how neural networks learn features, or relevant patterns in data, for prediction is necessary for their reliable use in technological and scientific applications. In this work, we presented a unifying mathematical mechanism, known as average gradient outer product (AGOP), that characterized feature learning in neural networks. We provided empirical evidence that AGOP captured features learned by various neural network architectures, including transformer-based language models, convolutional networks, multilayer perceptrons, and recurrent neural networks. Moreover, we demonstrated that AGOP, which is backpropagation-free, enabled feature learning in machine learning models, such as kernel machines, that a priori could not identify task-specific features. Overall, we established a fundamental mechanism that captured feature learning in neural networks and enabled feature learning in general machine learning models.