Convergence of Dynamic Programming on the Semidefinite Cone for Discrete-Time Infinite-Horizon LQR

• 成果类型: Article
• 署名作者: Lee, Donghwan
• 署名单位: Korea Advanced Institute of Science & Technology (KAIST)
• 刊物名称: IEEE TRANSACTIONS ON AUTOMATIC CONTROL
• ISSN/ISSBN: 0018-9286
• DOI: 10.1109/TAC.2022.3181752
• 发表日期: 2022
• 页码: 5661-5668
• 关键词: convergence; dynamic programming; linear time-invariant (LTI) system; optimal control; Reinforcement Learning

摘要

The goal of this article is to investigate new and simple convergence analysis of dynamic programming for the linear-quadratic regulator problem of discrete-time linear time-invariant systems. In particular, bounds on errors are given in terms of both matrix inequalities and matrix norm. Under a mild assumption on the initial parameter, we prove that the Q-value iteration exponentially converges to the optimal solution. Moreover, a global asymptotic convergence is also presented. These results are then extended to the policy iteration. We prove that in contrast to the Q-value iteration, the policy iteration always converges exponentially fast. An example is given to illustrate the results.