Structural Estimation of Markov Decision Processes in High Dimensional State Space with Finite-Time Guarantees

• 成果类型: Article
• 署名作者: Zeng, Siliang; Hong, Mingyi; Garcia, Alfredo
• 署名单位: University of Minnesota System; University of Minnesota Twin Cities; Texas A&M University System; Texas A&M University College Station
• 刊物名称: OPERATIONS RESEARCH
• ISSN/ISSBN: 0030-364X
• DOI: 10.1287/opre.2022.0511
• 发表日期: 2025
• 关键词: dynamic-models

摘要

We consider the task of estimating a structural model of dynamic decisions by human agent based on the observable history of implemented actions and visited states. This problem has an inherent nested structure: In the inner problem, an optimal policy a given reward function is identified, whereas in the outer problem, a measure of fit is maximized. Several approaches have been proposed to alleviate the computational burden this nested-loop structure, but these methods still suffer from high complexity when state space is either discrete with large cardinality or continuous in high dimensions. Other approaches in the inverse reinforcement learning literature emphasize policy estimation the expense of reduced reward estimation accuracy. In this paper, we propose a single-loop estimation algorithm with finite time guarantees that is equipped to deal with high dimensional state spaces without compromising reward estimation accuracy. In the proposed algorithm, each policy improvement step is followed by a stochastic gradient step for likelihood maximization. We show the proposed algorithm converges to a stationary solution with a finite-time guarantee. Further, if the reward is parameterized linearly, algorithm approximates the maximum likelihood estimator sublinearly.