Analysis of Gradient Descent With Varying Step Sizes Using Integral Quadratic Constraints

• 成果类型: Article
• 署名作者: Padmanabhan, Ram; Seiler, Peter
• 署名单位: University of Illinois System; University of Illinois Urbana-Champaign; University of Illinois System; University of Illinois Urbana-Champaign; University of Michigan System; University of Michigan
• 刊物名称: IEEE TRANSACTIONS ON AUTOMATIC CONTROL
• ISSN/ISSBN: 0018-9286
• DOI: 10.1109/TAC.2024.3438808
• 发表日期: 2025
• 页码: 587-594
• 关键词: NOISE; CONVERGENCE; optimization; Linear systems; gradient methods; Heuristic algorithms; Convex functions; Convex Optimization; Gradient descent; integral quadratic constraints (IQCs); Linear matrix inequalities (LMIs); linear parameter-varying (LPV) systems; varying step sizes

摘要

The framework of integral quadratic constraints is used to perform an analysis of gradient descent with varying step sizes. Two performance metrics are considered: convergence rate and noise amplification. We assume that the step size is produced from a line search and varies in a known interval. Modeling the algorithm as a linear parameter-varying (LPV) system, we construct a parameterized linear matrix inequality condition that certifies algorithm performance, which is solved using a result for polytopic LPV systems. Our results provide convergence rate guarantees when the step size lies within a restricted interval. Moreover, we recover existing rate bounds when this interval reduces to a single point, i.e., a constant step size. Finally, we note that the convergence rate depends only on the condition number of the problem. In contrast, the noise amplification performance depends on the individual values of the strong convexity and smoothness parameters, and varies inversely with them for a fixed condition number.