A Complete Set of Quadratic Constraints for Repeated ReLU and Generalizations

• 成果类型: Article
• 署名作者: Noori, Sahel Vahedi; Hu, Bin; Dullerud, Geir; Seiler, Peter
• 署名单位: University of Michigan System; University of Michigan; University of Illinois System; University of Illinois Urbana-Champaign; University of Minnesota System; University of Minnesota Twin Cities
• 刊物名称: IEEE TRANSACTIONS ON AUTOMATIC CONTROL
• ISSN/ISSBN: 0018-9286
• DOI: 10.1109/TAC.2025.3640487
• 发表日期: 2026
• 关键词: dissipative dynamical-systems; robustness analysis; linear-systems; stability analysis; neural-networks; IQC; performance; SUBJECT; DESIGN

摘要

This article derives a complete set of quadratic constraints (QCs) for the repeated ReLU. The complete set of QCs is described by a collection of matrix copositivity conditions. We also show that only two functions satisfy all QCs in our complete set: the repeated rectified linear unit (ReLU) and flipped ReLU. Thus, our complete set of QCs bounds the repeated ReLU as tight as possible up to the sign invariance inherent in quadratic forms. We derive a similar complete set of incremental QCs for repeated ReLU, which can potentially lead to less conservative Lipschitz bounds for ReLU networks than the standard LipSDP approach. The basic constructions are also used to derive the complete sets of QCs for other piecewise linear activation functions, such as leaky ReLU, MaxMin, and HouseHolder. Finally, we illustrate the use of the complete set of QCs to assess stability and performance for recurrent neural networks with ReLU activation functions. We rely on a standard copositivity relaxation to formulate the stability/performance condition as a semidefinite program. Simple examples are provided to illustrate that the complete sets of QCs and incremental QCs can yield less conservative bounds than existing sets.