Regularized Q-Learning With Linear Function Approximation
成果类型：
Article
署名作者：
Xi, Jiachen; Garcia, Alfredo; Momcilovic, Petar
署名单位：
Texas A&M University System; Texas A&M University College Station
刊物名称：
IEEE TRANSACTIONS ON AUTOMATIC CONTROL
ISSN/ISSBN：
0018-9286
DOI：
10.1109/TAC.2025.3592801
发表日期：
2026
关键词：
摘要：
We consider a single-loop algorithm for regularized Q-learning with linear function approximation. The proposed algorithm is motivated by a bilevel optimization formulation of regularized Q-learning wherein the lower level optimization problem aims to identify a value function approximation that satisfies Bellman's recursive optimality condition, and the upper level aims to find the projection onto the span of basis vectors. We show that under certain assumptions, the proposed algorithm converges to a stationary point in the presence of Markovian noise. In addition, we provide a performance guarantee for the policies derived from the proposed algorithm.