Heterogeneity, reinforcement learning, and chaos in population games

• 成果类型: Article
• 署名作者: Bielawski, Jakub; Chotibut, Thiparat; Falniowski, Fryderyk; Misiurewicz, Michal; Piliouras, Georgios
• 署名单位: Cracow University of Economics; Chulalongkorn University; Indiana University System; Indiana University Indianapolis; Alphabet Inc.; Google Incorporated; DeepMind
• 刊物名称: PROCEEDINGS OF THE NATIONAL ACADEMY OF SCIENCES OF THE UNITED STATES OF AMERICA
• ISSN/ISSBN: 0027-8954
• DOI: 10.1073/pnas.2319929121
• 发表日期: 2025-06-24
• 关键词: evolutionary dynamics; EFFICIENCY; selection; discrete; models

摘要

Inspired by the challenges at the intersection of Evolutionary Game Theory and Machine Learning, we investigate a class of discrete-time multiagent reinforcement learning (MARL) dynamics in population/nonatomic congestion games, where agents have diverse beliefs and learn at different rates. These congestion games, a well-studied class of potential games, are characterized by individual agents having negligible effects on system performance, strongly aligned incentives, and well-understood advantageous properties of Nash equilibria. Despite the presence of static Nash equilibria, we demonstrate that MARL dynamics with heterogeneous learning rates can deviate from these equilibria, exhibiting instability and even chaotic behavior and resulting in increased social costs. Remarkably, even within these chaotic regimes, we show that the time-averaged macroscopic behavior converges to exact Nash equilibria, thus linking the microscopic dynamic complexity with traditional equilibrium concepts. By employing dynamical systems techniques, we analyze the interaction between individual-level adaptation and population-level outcomes, paving the way for studying heterogeneous learning dynamics in discrete time across more complex game scenarios.