Gradient Play in Stochastic Games: Stationary Points, Convergence, and Sample Complexity

• 成果类型: Article
• 署名作者: Zhang, Runyu; Ren, Zhaolin; Li, Na
• 署名单位: Harvard University
• 刊物名称: IEEE TRANSACTIONS ON AUTOMATIC CONTROL
• ISSN/ISSBN: 0018-9286
• DOI: 10.1109/TAC.2024.3387208
• 发表日期: 2024
• 页码: 6499-6514
• 关键词: games; CONVERGENCE; Stochastic processes; Nash equilibrium; geometry; Complexity theory; stability analysis; multiagent systems; Reinforcement Learning

摘要

In this article, we study the performance of the gradient play algorithm for stochastic games (SGs), where each agent tries to maximize its own total discounted reward by making decisions independently based on current state information, which is shared between agents. Policies are directly parameterized by the probability of choosing a certain action at a given state. We show that Nash equilibria (NEs) and first-order stationary policies are equivalent in this setting, and give a local convergence rate around strict NEs. Furthermore, for a subclass of SGs called Markov potential games (which includes the setting with identical rewards as an important special case), we design a sample-based reinforcement learning algorithm and give a nonasymptotic global convergence rate analysis for both exact gradient play and our sample-based learning algorithm. Our result shows that the number of iterations to reach an epsilon-NE scales linearly, instead of exponentially, with the number of agents. Local geometry and local stability are also considered, where we prove that strict NEs are local maxima of the total potential function and fully mixed NEs are saddle points.