Asynchronous Stochastic Approximations With Asymptotically Biased Errors and Deep Multiagent Learning

• 成果类型: Article
• 署名作者: Ramaswamy, Arunselvan; Bhatnagar, Shalabh; Quevedo, Daniel E.
• 署名单位: University of Paderborn; University of Paderborn; Indian Institute of Science (IISC) - Bangalore; Bosch; Indian Institute of Science (IISC) - Bangalore; Queensland University of Technology (QUT)
• 刊物名称: IEEE TRANSACTIONS ON AUTOMATIC CONTROL
• ISSN/ISSBN: 0018-9286
• DOI: 10.1109/TAC.2020.3026269
• 发表日期: 2021
• 页码: 3969-3983
• 关键词: approximation algorithms; Approximation Error; stability analysis; Function approximation; Stochastic processes; asymptotic stability; CONVERGENCE; Almost sure boundedness (stability); asymptotically biased approximation errors; asynchronous stochastic approximations; deep function approximations; Deep Reinforcement Learning; Distributed control; multi agent learning; networked control systems; neuro-dynamic programming

摘要

Asynchronous stochastic approximations (SAs) are an important class of model-free algorithms, tools, and techniques that are popular in multiagent and distributed control scenarios. To counter Bellman's curse of dimensionality, such algorithms are coupled with function approximations. Although the learning/control problem becomes more tractable, function approximations affect stability and convergence. In this article, we present verifiable sufficient conditions for stability and convergence of asynchronous SAs with biased approximation errors. The theory developed herein is used to analyze policy gradient methods and noisy value iteration schemes. Specifically, we analyze the asynchronous approximate counterparts of the policy gradient (A2PG) and value iteration (A2VI) schemes. It is shown that the stability of these algorithms is unaffected by biased approximation errors, provided that they are asymptotically bounded. With respect to convergence (of A2VI and A2PG), a relationship between the limiting set and the approximation errors is established. Finally, experimental results that support the theory are presented.