Maximum Entropy Optimal Control of Continuous-Time Dynamical Systems

• 成果类型: Article
• 署名作者: Kim, Jeongho; Yang, Insoon
• 署名单位: Seoul National University (SNU); Korea Institute for Advanced Study (KIAS); Seoul National University (SNU); Seoul National University (SNU)
• 刊物名称: IEEE TRANSACTIONS ON AUTOMATIC CONTROL
• ISSN/ISSBN: 0018-9286
• DOI: 10.1109/TAC.2022.3168168
• 发表日期: 2023
• 页码: 2018-2033
• 关键词: Dynamic programming (DP); entropy; Hamilton-Jacobi-Bellman (HJB) equations; optimal control; Viscosity solution

摘要

Maximum entropy reinforcement learning methods have been successfully applied to a range of challenging sequential decision-making and control tasks. However, most of the existing techniques are designed for discrete-time systems although there has been a growing interest to handle physical processes evolving in continuous time. As a first step toward their extension to continuous-time systems, this article aims to study the theory of maximum entropy optimal control in continuous time. Applying the dynamic programming principle, we derive a novel class of Hamilton-Jacobi-Bellman (HJB) equations and prove that the optimal value function of the maximum entropy control problem corresponds to the unique viscosity solution of the HJB equation. We further show that the optimal control is uniquely characterized as Gaussian in the case of control-affine systems and that, for linear-quadratic problems, the HJB equation is reduced to a Riccati equation, which can be used to obtain an explicit expression of the optimal control. The results of our numerical experiments demonstrate the performance of our maximum entropy method in continuous-time optimal control and reinforcement learning problems.