Universal Approximation Power of Deep Residual Neural Networks Through the Lens of Control

• 成果类型: Article
• 署名作者: Tabuada, Paulo; Gharesifard, Bahman
• 署名单位: University of California System; University of California Los Angeles
• 刊物名称: IEEE TRANSACTIONS ON AUTOMATIC CONTROL
• ISSN/ISSBN: 0018-9286
• DOI: 10.1109/TAC.2022.3190051
• 发表日期: 2023
• 页码: 2715-2728
• 关键词: Control systems; CONTROLLABILITY; Biological neural networks; Residual neural networks; neurons; control theory; Differential equations; Neural Networks; nonlinear controllability; residual networks; universal approximation

摘要

In this article, we show that deep residual neural networks have the power of universal approximation by using, in an essential manner, the observation that these networks can be modeled as nonlinear control systems. We first study the problem of using a deep residual neural network to exactly memorize training data by formulating it as a controllability problem for an ensemble control system. Using techniques from geometric control theory, we identify a class of activation functions that allow us to ensure controllability on an open and dense submanifold of sample points. Using this result, and resorting to the notion of monotonicity, we establish that any continuous function can be approximated on a compact set to arbitrary accuracy, with respect to the uniform norm, by this class of neural networks. Moreover, we provide optimal bounds on the number of required neurons.