Global Optimality Guarantees for Policy Gradient Methods

• 成果类型: Article
• 署名作者: Bhandari, Jalaj; Russo, Daniel
• 署名单位: Columbia University; Columbia University
• 刊物名称: OPERATIONS RESEARCH
• ISSN/ISSBN: 0030-364X
• DOI: 10.1287/opre.2021.0014
• 发表日期: 2024
• 关键词: base-stock levels; optimization; bounds; simulation; Iteration; DESIGN

摘要

Policy gradients methods apply to complex, poorly understood, control problems by performing stochastic gradient descent over a parameterized class of polices. Unfortunately, even for simple control problems solvable by standard dynamic programming techniques, policy gradient algorithms face nonconvex optimization problems and are widely understood to converge only to a stationary point. This work identifies structural properties, shared by several classic control problems, that ensure the policy gradient objective function has no suboptimal stationary points despite being nonconvex. When these conditions are strengthened, this objective satisfies a Polyak-lojasiewicz (gradient dominance) condition that yields convergence rates. We also provide bounds on the optimality gap of any stationary point when some of these conditions are relaxed.