Easy Affine Markov Decision Processes

• 成果类型: Article
• 署名作者: Ning, Jie; Sobel, Matthew J.
• 署名单位: University System of Ohio; Case Western Reserve University
• 刊物名称: OPERATIONS RESEARCH
• ISSN/ISSBN: 0030-364X
• DOI: 10.1287/opre.2018.1836
• 发表日期: 2019
• 页码: 1719-1737
• 关键词: myopic solutions; bounds; AGE; fisheries; policies; fish

摘要

This paper characterizes the class of decomposable affine Markov decision processes (MDPs), which have continuous multidimensional endogenous states and actions, and Markov-modulated exogenous states. This class of MDPs has affine dynamics and single-period rewards, sets of feasible actions that decompose into bounded polytopes, and endogenous state variables that are nonnegative or nonpositive. It is shown that decomposable affine MDPs with discounted criteria have an affine value function and an affine optimal policy. The affine coefficients of the value function and optimal policy are determined by the solution of auxiliary equations, which themselves resemble the dynamic program of a finite MDP. This result exorcizes the curse of dimensionality for decomposable affine MDPs, which otherwise could be solved only approximately with discrete approximations. Additionally, the paper characterizes partially decomposable affine MDPs that meet only some of the assumptions for decomposable affine MDPs. It shows that they are composites of two smaller MDPs, one of which is a decomposable affine MDP. The applicability of the classes of MDPs in the paper is exemplified with models of fishery management, dynamic capacity portfolio management, and commodity procurement.