On Exact Embedding Framework for Optimal Control of Markov Decision Processes

• 成果类型: Article
• 署名作者: Kharade, Sonam; Sutavani, Sarang; Yerudkar, Amol; Wagh, Sushama; Liu, Yang; Del Vecchio, Carmen; Singh, N. M.
• 署名单位: University of South Carolina System; University of South Carolina Columbia; Clemson University; Zhejiang Normal University; Veermata Jijabai Technological Institute (VJTI); University of Sannio
• 刊物名称: IEEE TRANSACTIONS ON AUTOMATIC CONTROL
• ISSN/ISSBN: 0018-9286
• DOI: 10.1109/TAC.2023.3301818
• 发表日期: 2024
• 页码: 1316-1323
• 关键词: Embedding; Kullback-Leibler (KL) divergence; linearly-solvable Markov decision processes (LMDPs); Markov decision processes (MDPs); optimal control

摘要

This article deals with the embedding framework of Markov decision processes (MDPs) with discrete state and action space to find optimal actions. The optimal control problem of MDPs can be efficiently tackled by restructuring the same into an equivalent linearly-solvable Markov decision processes (LMDPs) through the method called embedding. However, state costs under the embedding may not exactly match the original costs and even assume unrealistic values. In this work, we derive a constructive sufficient condition to devise an exact embedding solution rendering the embedded state cost to match the original system. Furthermore, since, in this case, the embedding implies a transition from the discrete to continuous action space, the correlation between the obtained continuous action and an equivalent desired discrete action is investigated using a maximum a posteriori probability-based method. Finally, some examples, including mammalian cell-cycle network, are presented to demonstrate the effectiveness of the proposed method.