Exponential convergence rates for momentum stochastic gradient descent in the overparametrized setting

• 成果类型: Article; Early Access
• 署名作者: Gess, Benjamin; Kassing, Sebastian
• 署名单位: University of Wuppertal; Technical University of Berlin; Max Planck Society
• 刊物名称: MATHEMATICAL PROGRAMMING
• ISSN/ISSBN: 0025-5610; 1436-4646
• DOI: 10.1007/s10107-025-02308-y
• 发表日期: 2026-01-07
• 关键词: Momentum stochastic gradient descent; PL-inequality; almost sure convergence; Overparametrization; Damping; differential-equations

摘要

We prove explicit bounds on the exponential rate of convergence for the momentum stochastic gradient descent scheme (MSGD) for arbitrary, fixed hyperparameters (learning rate, friction parameter) and its continuous-in-time counterpart in the context of non-convex optimization. The results are shown for objective functions satisfying a local Polyak-& Lstrok;ojasiewicz inequality and under assumptions on the variance of MSGD that are satisfied in overparametrized settings. Moreover, we analyze the optimal choice of the friction parameter and show that the MSGD process almost surely converges to a local minimum.