Policy Optimization for Markovian Jump Linear Quadratic Control: Gradient Method and Global Convergence

• 成果类型: Article
• 署名作者: Jansch-Porto, Joao Paulo; Hu, Bin; Dullerud, Geir E.
• 署名单位: University of Illinois System; University of Illinois Urbana-Champaign; University of Illinois System; University of Illinois Urbana-Champaign
• 刊物名称: IEEE TRANSACTIONS ON AUTOMATIC CONTROL
• ISSN/ISSBN: 0018-9286
• DOI: 10.1109/TAC.2022.3176439
• 发表日期: 2023
• 页码: 2475-2482
• 关键词: Optimization; Linear systems; gradient methods; CONVERGENCE; State feedback; COSTS; Markov processes; Markovian jump linear systems (MJLS); optimal control; policy gradient methods; reinforcement learning (RL)

摘要

Recently, policy optimization has received renewed attention from the control community due to various applications in reinforcement learning tasks. In this article, we investigate the global convergence of the gradient method for quadratic optimal control of discrete-time Markovian jump linear systems (MJLS). First, we study the optimization landscape of direct policy optimization for MJLS, with static-state feedback controllers and quadratic performance costs. Despite the nonconvexity of the resultant problem, we are still able to identify several useful properties such as coercivity, gradient dominance, and smoothness. Based on these properties, we prove that the gradient method converges to the optimal-state feedback controller for MJLS at a linear rate if initialized at a controller, which is mean-square stabilizing. This article brings new insights for understanding the performance of the policy gradient method on the Markovian jump linear quadratic control problem.