A General Framework for Learning Mean-Field Games

• 成果类型: Article
• 署名作者: Guo, Xin; Hu, Anran; Xu, Renyuan; Zhang, Junzi
• 署名单位: University of California System; University of California Berkeley; Amazon.com; University of Southern California; University of Oxford; Stanford University
• 刊物名称: MATHEMATICS OF OPERATIONS RESEARCH
• ISSN/ISSBN: 0364-765X
• DOI: 10.1287/moor.2022.1274
• 发表日期: 2023
• 页码: 656-686
• 关键词: convergence

摘要

This paper presents a general mean-field game (GMFG) framework for simultaneous learning and decision making in stochastic games with a large population. It first establishes the existence of a unique Nash equilibrium to this GMFG, and it demonstrates that naively combining reinforcement learning with the fixed-point approach in classical mean-field games yields unstable algorithms. It then proposes value-based and policy-based reinforcement learning algorithms (GMF-V and GMF-P, respectively) with smoothed policies, with analysis of their convergence properties and computational complexities. Experiments on an equilibrium product pricing problem demonstrate that two specific instantiations of GMF-V with Q-learning and GMF-P with trust region policy optimization-GMF-V-Q and GMF-P-TRPO, respectively-are both efficient and robust in the GMFG setting. Moreover, their performance is superior in convergence speed, accuracy, and stability when compared with existing algorithms for multiagent reinforcement learning in the N-player setting.