Distributionally Robust Markov Decision Processes and Their Connection to Risk Measures

• 成果类型: Article; Early Access
• 署名作者: Baeuerle, Nicole; Glauner, Alexander
• 署名单位: Helmholtz Association; Karlsruhe Institute of Technology
• 刊物名称: MATHEMATICS OF OPERATIONS RESEARCH
• ISSN/ISSBN: 0364-765X
• DOI: 10.1287/moor.2021.1187
• 发表日期: 2021
• 关键词: Ambiguity; MODEL

摘要

We consider robust Markov decision processes with Borel state and action spaces, unbounded cost, and finite time horizon. Our formulation leads to a Stackelberg game against nature. Under integrability, continuity, and compactness assumptions, we derive a robust cost iteration for a fixed policy of the decision maker and a value iteration for the robust optimization problem. Moreover, we show the existence of deterministic optimal policies for both players. This is in contrast to classical zero-sum games. In case the state space is the real line, we show under some convexity assumptions that the interchange of supremum and infimum is possible with the help of Sion's minimax theorem. Further, we consider the problem with special ambiguity sets. In particular, we are able to derive some cases where the robust optimization problem coincides with the minimization of a coherent risk measure. In the final section, we discuss two applications: a robust linear quadratic problem and a robust problem for managing regenerative energy.