Distributional Reachability for Markov Decision Processes: Theory and Applications

• 成果类型: Article
• 署名作者: Gao, Yulong; Abate, Alessandro; Xie, Lihua; Johansson, Karl Henrik
• 署名单位: Imperial College London; University of Oxford; Nanyang Technological University
• 刊物名称: IEEE TRANSACTIONS ON AUTOMATIC CONTROL
• ISSN/ISSBN: 0018-9286
• DOI: 10.1109/TAC.2023.3341282
• 发表日期: 2024
• 页码: 4598-4613
• 关键词: Distributional reachability; Markov decision processes (MDPs); probabilistic reachability; reach-avoid problems; set invariance

摘要

We study distributional reachability for finite Markov decision processes (MDPs) from a control theoretical perspective. Unlike standard probabilistic reachability notions, which are defined over MDP states or trajectories, in this article reachability is formulated over the space of probability distributions. We propose two set-valued maps for the forward and backward distributional reachability problems: the forward map collects all state distributions that can be reached from a set of initial distributions, while the backward map collects all state distributions that can reach a set of final distributions. We show that there exists a maximal invariant set under the forward map and this set is the region where the state distributions eventually always belong to, regardless of the initial state distribution and policy. The backward map provides an alternative way to solve a class of important problems for MDPs: the study of controlled invariance, the characterization of the domain of attraction, and reach-avoid problems. Three case studies illustrate the effectiveness of our approach.