Continuous-time controlled Markov chains

• 成果类型: Article
• 署名作者: Guo, XP; Hernández-Lerma, O
• 署名单位: Sun Yat Sen University; CINVESTAV - Centro de Investigacion y de Estudios Avanzados del Instituto Politecnico Nacional; Instituto Politecnico Nacional - Mexico
• 刊物名称: ANNALS OF APPLIED PROBABILITY
• ISSN/ISSBN: 1050-5164
• 发表日期: 2003
• 页码: 363-388
• 关键词: DECISION-PROCESSES; bias optimality; countable state; models; birth

摘要

This paper concerns studies on continuous-time controlled Markov chains, that is, continuous-time Markov decision processes with a denumerable state space, with respect to the discounted cost criterion. The cost and transition rates are allowed to be unbounded and the action set is a Borel space. We first study control problems in the class of deterministic stationary policies and give very weak conditions under which the existence of epsilon-optimal (epsilon greater than or equal to 0) policies is proved using the construction of a minimum Q-process. Then we further consider control problems in the class of randomized Markov policies for (1) regular and (2) nonregular Q-processes. To study case (1), first we present a new necessary and sufficient condition for a nonhomogeneous Q-process to be regular. This regularity condition, together with the extended generator of a nonhomogeneous Markov process, is used to prove the existence of epsilon-optimal stationary policies. Our results for case (1) are illustrated by a Schlogl model with a controlled diffusion. For case (2), we obtain a similar result using Kolmogorov's forward equation for the minimum Q-process and we also present an example in which our assumptions are satisfied, but those used in the previous literature fail to hold.