mlsanityData to 14 Oct 2026

Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized Multi-Agent Reinforcement Learning

arXiv 2505.18433 ↗First posted 24 May 2025cs.LG cs.MAcited by 0

Not on the map. Why?

Abstract (arXiv)

Actor-critic methods for decentralized multi-agent reinforcement learning (MARL) facilitate collaborative optimal decision making without centralized coordination, thus enabling a wide range of applications in practice. To date, however, most theoretical convergence studies for existing actor-critic decentralized MARL methods are limited to the guarantee of a stationary solution under the linear function approximation. This leaves a significant gap between the highly successful use of deep neural actor-critic for decentralized MARL in practice and the current theoretical understanding. To bridge this gap, in this paper, we make the first attempt to develop a deep neural actor-critic method for decentralized MARL, where both the actor and critic components are inherently non-linear. We show that our proposed method enjoys a global optimality guarantee with a finite-time convergence rate of O(1/T), where T is the total iteration times. This marks the first global convergence result for deep neural actor-critic methods in the MARL literature. We also conduct extensive numerical experiments, which verify our theoretical results.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 41 of its 67 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

41 of 67 references.

References · 41

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  5. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  6. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  7. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  9. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  11. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 575
  12. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
29 more
  1. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  3. 2017Emergence of Grounded Compositional Language in Multi-Agent Populationscited by 263
  4. 2023Large Language Model Alignment: A Surveycited by 243
  5. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 225
  6. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 194
  7. 2018Applications of Deep Reinforcement Learning in Communications and Networking: A Surveycited by 167
  8. 2017Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observabilitycited by 148
  9. 2021A general sample complexity analysis of vanilla policy gradientcited by 76
  10. 2018Multi-Agent Deep Reinforcement Learning for Dynamic Power Allocation in Wireless Networkscited by 70
  11. 2020Single and Multi-Agent Deep Reinforcement Learning for AI-Enabled Wireless Networks: A Tutorialcited by 60
  12. 2019Convergence of Adversarial Training in Overparametrized Neural Networkscited by 58
  13. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 58
  14. 2018Multi-Agent Reinforcement Learning Based Resource Allocation for UAV Networkscited by 56
  15. 2020Federated Bandit: A Gossiping Approachcited by 55
  16. 2020Non-asymptotic Convergence Analysis of Two Time-scale (Natural) Actor-Critic Algorithmscited by 43
  17. 2020Resource Management in Wireless Networks via Multi-Agent Deep Reinforcement Learningcited by 42
  18. 2018Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimizationcited by 40
  19. 2015Decentralized Q-Learning for Stochastic Teams and Gamescited by 35
  20. 2021Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Surveycited by 34
  21. 2021Sample and Communication-Efficient Decentralized Actor-Critic Algorithms with Finite-Time Analysiscited by 26
  22. 2021On the Global Optimum Convergence of Momentum-based Policy Gradientcited by 26
  23. 2015Fully Connected Deep Structured Networkscited by 24
  24. 2022A2C is a special case of PPOcited by 16
  25. 2020Multi-agent Reinforcement Learning for Resource Allocation in IoT networks with Edge Computingcited by 13
  26. 2018Convergent Actor-Critic Algorithms Under Off-Policy Training and Function Approximationcited by 12
  27. 2021Learning to Coordinate in Multi-Agent Systems: A Coordinated Actor-Critic Algorithm and Finite-Time Guaranteescited by 6
  28. 2023On the Global Convergence of Natural Actor-Critic with Two-layer Neural Network Parametrizationcited by 5
  29. 2024Sample and Communication Efficient Fully Decentralized MARL Policy Evaluation via a New Approach: Local TD updatecited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025A Communication-Efficient Decentralized Actor-Critic Algorithm
  2. 2026Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic
  3. 2026Global Convergence of Average Reward Constrained MDPs with Neural Critic and General Policy Parameterization
  4. 2024On Centralized Critics in Multi-Agent Reinforcement Learning
  5. 2024Natural Policy Gradient and Actor Critic Methods for Constrained Multi-Task Reinforcement Learning
  6. 2026Provably Convergent Actor-Critic for MARL through Risk-aversion
  7. 2025Fully-Decentralized MADDPG with Networked Agents
  8. 2024Convergence of Decentralized Actor-Critic Algorithm in General-sum Markov Games