On Passivity, Reinforcement Learning, and Higher Order Learning in Multiagent Finite Games

• 成果类型: Article
• 署名作者: Gao, Bolin; Pavel, Lacra
• 署名单位: University of Toronto
• 刊物名称: IEEE TRANSACTIONS ON AUTOMATIC CONTROL
• ISSN/ISSBN: 0018-9286
• DOI: 10.1109/TAC.2020.2978037
• 发表日期: 2021
• 页码: 121-136
• 关键词: games; CONVERGENCE; Learning (artificial intelligence); Heuristic algorithms; sociology; statistics; Radio frequency; Agents and autonomous systems; game theory; Nonlinear systems; Passivity; Reinforcement Learning

摘要

In this article, we propose a passivity-based methodology for the analysis and design of reinforcement learning dynamics and algorithms in multiagent finite games. Starting from a known, first-order reinforcement learning scheme, we show that convergence to a Nash distribution can be attained in a broader class of games than previously considered in the literature-namely, in games characterized by the monotonicity property of their (negative) payoff vectors. We further exploit passivity techniques to design a class of higher order learning schemes that preserve the convergence properties of their first-order counterparts. Moreover, we show that the higher order schemes improve upon the rate of convergence and can even achieve convergence where the first-order scheme fails. We demonstrate these properties through numerical simulations for several representative games.