mlsanityData to 7 Oct 2026

The Gradient Convergence Bound of Federated Multi-Agent Reinforcement Learning with Efficient Communication

arXiv 2103.13026 ↗First posted 24 Mar 2021cs.LG cs.AI cs.MAcited by 5

Not on the map. Why?

Abstract (arXiv)

The paper considers independent reinforcement learning (IRL) for multi-agent collaborative decision-making in the paradigm of federated learning (FL). However, FL generates excessive communication overheads between agents and a remote central server, especially when it involves a large number of agents or iterations. Besides, due to the heterogeneity of independent learning environments, multiple agents may undergo asynchronous Markov decision processes (MDPs), which will affect the training samples and the model's convergence performance. On top of the variation-aware periodic averaging (VPA) method and the policy-based deep reinforcement learning (DRL) algorithm (i.e., proximal policy optimization (PPO)), this paper proposes two advanced optimization schemes orienting to stochastic gradient descent (SGD): 1) A decay-based scheme gradually decays the weights of a model's local gradients with the progress of successive local updates, and 2) By representing the agents as a graph, a consensus-based scheme studies the impact of exchanging a model's local gradients among nearby agents from an algebraic connectivity perspective. This paper also provides novel convergence guarantees for both developed schemes, and demonstrates their superior effectiveness and efficiency in improving the system's utility value through theoretical analyses and simulation results.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 41 of its 90 references that are in the map. First posted on arXiv 24 Mar 2021.

Builds on

41 of 90 references.

References · 41

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  7. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  10. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  11. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  12. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
29 more
  1. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  2. 2019Federated Learning over Wireless Fading Channelscited by 178
  3. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  4. 2017Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observabilitycited by 143
  5. 2013Mini-batch Stochastic Approximation Methods for Nonconvex Stochastic Composite Optimizationcited by 142
  6. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 135
  7. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 115
  8. 2018Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithmscited by 106
  9. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 106
  10. 2017On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimizationcited by 91
  11. 2014Deep learning with Elastic Averaging SGDcited by 90
  12. 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
  13. 2016CoCoA: A General Framework for Communication-Efficient Distributed Optimizationcited by 69
  14. 2018Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGDcited by 68
  15. 2020Towards Flexible Device Participation in Federated Learningcited by 64
  16. 2017Collaborative Deep Learning in Fixed Topology Networkscited by 60
  17. 2019DeepPool: Distributed Model-free Algorithm for Ride-sharing using Deep Reinforcement Learningcited by 54
  18. 2019Federated Transfer Reinforcement Learning for Autonomous Drivingcited by 43
  19. 2018On the Fenchel Duality between Strong Convexity and Lipschitz Continuous Gradientcited by 42
  20. 2020Multi-Stage Hybrid Federated Learning over Large-Scale D2D-Enabled Fog Networkscited by 42
  21. 2020Decentralized Federated Learning via SGD over Wireless D2D Networkscited by 32
  22. 2020SQuARM-SGD: Communication-Efficient Momentum SGD for Decentralized Optimizationcited by 31
  23. 2019Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learningcited by 18
  24. 2020Proxy Experience Replay: Federated Distillation for Distributed Reinforcement Learningcited by 10
  25. 2019Stigmergic Independent Reinforcement Learning for Multi-Agent Collaborationcited by 8
  26. 2020Concentrated Differentially Private and Utility Preserving Federated Learningcited by 5
  27. 2020Differentially Private Federated Learning for Resource-Constrained Internet of Thingscited by 4
  28. 2020Local Stochastic Approximation: A Unified View of Federated Learning and Distributed Multi-Task Reinforcement Learning Algorithmscited by 3
  29. 2017Parle: parallelizing stochastic gradient descentcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 0; 2025: 3.

5 papers.

  1. 2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
  2. 2025Tool-Aided Evolutionary LLM for Generative Policy Toward Efficient Resource Management in Wireless Federated Learningcited by 2
  3. 2025Robust and Efficient Communication in Multi-Agent Reinforcement Learningcited by 2
  4. 2025Provable Performance Bounds for Digital Twin-driven Deep Reinforcement Learning in Wireless Networks: A Novel Digital-Twin Bisimulation Metriccited by 1
  5. 2023Over-the-air Federated Policy Gradientcited by 0