Solution and forecast horizons for infinite-horizon nonhornogeneous Markov decision processes

• 成果类型: Article
• 署名作者: Cheevaprawatdomrong, Torpong; Schochetman, Irwin E.; Smith, Robert L.; Garcia, Alfredo
• 署名单位: Oakland University; University of Michigan System; University of Michigan; University of Virginia
• 刊物名称: MATHEMATICS OF OPERATIONS RESEARCH
• ISSN/ISSBN: 0364-765X
• DOI: 10.1287/moor.1060.0224
• 发表日期: 2007
• 页码: 51-72
• 关键词: Time-varying systems; optimization problems; planning-horizons; EXISTENCE; RULE

摘要

We consider a nonhomogeneous infinite-horizon Markov Decision Process (MDP) problem with multiple optimal first-period policies. We seek an algorithm that, given finite data, delivers an optimal first-period policy. Such an algorithm can thus recursively generate, within a rolling-horizon procedure, an infinite-horizon optimal solution to the original problem. However, it can happen that no such algorithm exists, i.e., the MDP is not well posed. Equivalently, it is impossible to solve the problem with a finite amount of data. Assuming increasing marginal returns in actions (with respect to states) and stochastically increasing state transitions (with respect to actions), we provide an algorithm that is guaranteed to solve the given MDP whenever it is well posed. This algorithm determines, in finite time, a forecast horizon for which an optimal solution delivers an optimal first-period policy. As an application, we solve all well-posed instances of the time-varying version of the classic asset-selling problem.