Risk-Averse Approximate Dynamic Programming with Quantile-Based Risk Measures

• 成果类型: Article
• 署名作者: Jiang, Daniel R.; Powell, Warren B.
• 署名单位: Pennsylvania Commonwealth System of Higher Education (PCSHE); University of Pittsburgh; Princeton University
• 刊物名称: MATHEMATICS OF OPERATIONS RESEARCH
• ISSN/ISSBN: 0364-765X
• DOI: 10.1287/moor.2017.0872
• 发表日期: 2018
• 页码: 554-579
• 关键词: multistage stochastic programs; Value-at-risk; optimization; CONVERGENCE; algorithms; simulation

摘要

In this paper, we consider a finite-horizon Markov decision process (MDP) for which the objective at each stage is to minimize a quantile-based risk measure (QBRM) of the sequence of future costs; we call the overall objective a dynamic quantile-based risk measure (DQBRM). In particular, we consider optimizing dynamic risk measures where the one-step risk measures are QBRMs, a class of risk measures that includes the popular value at risk (VaR) and the conditional value at risk (CVaR). Although there is considerable theoretical development of risk-averse MDPs in the literature, the computational challenges have not been explored as thoroughly. We propose data-driven and simulation-based approximate dynamic programming (ADP) algorithms to solve the risk-averse sequential decision problem. We address the issue of inefficient sampling for risk applications in simulated settings and present a procedure, based on importance sampling, to direct samples toward the risky region as the ADP algorithm progresses. Finally, we show numerical results of our algorithms in the context of an application involving risk-averse bidding for energy storage.