A DYNAMIC ANALYTIC METHOD FOR RISK-AWARE CONTROLLED MARTINGALE PROBLEMS

• 成果类型: Article
• 署名作者: Isohatala, Jukka; Haskell, William B.
• 署名单位: National University of Singapore; Purdue University System; Purdue University
• 刊物名称: ANNALS OF APPLIED PROBABILITY
• ISSN/ISSBN: 1050-5164
• DOI: 10.1214/22-AAP1794
• 发表日期: 2023
• 页码: 1661-1700
• 关键词: markov decision-processes; jacobi-bellman equations; time-average control; diffusion-processes; EXISTENCE; PRINCIPLE

摘要

We present a new, tractable method for solving risk-aware problems over finite and infinite, discounted time-horizons where the dynamics of the con-trolled process are described using the martingale method. Supposing gen-eral Polish state and action spaces, and using the martingale characteriza-tion, we state a risk-aware dynamic optimal control problem of minimizing risk of costs described by a generic risk function. From this, we construct an alternative formulation of the optimization problem that takes the form of a nonlinear programming problem, constrained by the dynamic, that is, time-dependent and linear Kolmogorov forward equation describing the time -dependent distribution of the state and running costs. This formulation is sim-ilar to the convex analytic method, in that the control problem is recast into a form where the objective is optimized over distributions representing the state space visitation frequencies. However, in our approach, the distributions are dynamic and also encode the cost distribution. As our main results, we prove the equivalence of the original martingale and dynamic analytic problems, in the sense that both have the same optimal values, and that the solution of either problem yields a solution of the other. Moreover, we find an optimal control process can be taken to be Markov in the controlled process state, running costs, and time. We further show that under additional assumptions the optimal value is attained. An example numeric problem is presented and solved.