Relaxed Equilibria for Time-Inconsistent Markov Decision Processes

• 成果类型: Article; Early Access
• 署名作者: Bayraktar, Erhan; Huang, Yu-Jui; Wang, Zhenhua; Zhou, Zhou
• 署名单位: University of Michigan System; University of Michigan; University of Colorado System; University of Colorado Boulder; Shandong University; Shandong University; University of Sydney
• 刊物名称: MATHEMATICS OF OPERATIONS RESEARCH
• ISSN/ISSBN: 0364-765X
• DOI: 10.1287/moor.2023.0209
• 发表日期: 2024
• 关键词: policies

摘要

This paper considers an infinite-horizon Markov decision process (MDP) that allows for general nonexponential discount functions in both discrete and continuous time. Because of the inherent time inconsistency, we look for a randomized equilibrium policy (i.e., relaxed equilibrium) in an intrapersonal game between an agent's current and future selves. When we modify the MDP by entropy regularization, a relaxed equilibrium is shown to exist by a nontrivial entropy estimate. As the degree of regularization diminishes, the entropy-regularized MDPs approximate the original MDP, which gives the general existence of a relaxed equilibrium in the limit by weak convergence arguments. As opposed to prior studies that consider only deterministic policies, our existence of an equilibrium does not require any convexity (or concavity) of the controlled transition probabilities and reward function. Interestingly, this benefit of considering randomized policies is unique to the time-inconsistent case.