mlsanityData to 14 Oct 2026

Global Convergence Guarantees for Federated Policy Gradient Methods with Adversaries

arXiv 2403.09940 ↗First posted 15 Mar 2024cs.LG cs.AI math.OCcited by 6

On the map: Deep reinforcement learning methods → RL for energy, networks and operations

Abstract (arXiv)

Federated Reinforcement Learning (FRL) allows multiple agents to collaboratively build a decision making policy without sharing raw trajectories. However, if a small fraction of these agents are adversarial, it can lead to catastrophic results. We propose a policy gradient based approach that is robust to adversarial agents which can send arbitrary values to the server. Under this setting, our results form the first global convergence guarantees with general parametrization. These results demonstrate resilience with adversaries, while achieving optimal sample complexity of order tildeOleft( frac1Nε2 left( 1+ fracf2Nright)right), where N is the total number of agents and f<N/2 is the number of adversarial agents.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods. Builds on 41 of its 61 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

41 of 61 references.

References · 41

  1. 2019Advances and Open Problems in Federated Learningcited by 3,046
  2. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  3. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  4. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 711
  5. 2018The Hidden Vulnerability of Distributed Learning in Byzantiumcited by 329
  6. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  7. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 306
  8. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 159
  9. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 143
  10. 2020Byzantine-Robust Learning on Heterogeneous Datasets via Bucketingcited by 128
  11. 2018Sever: A Robust Meta-Algorithm for Stochastic Optimizationcited by 125
  12. 2018Stochastic Variance-Reduced Policy Gradientcited by 106
29 more
  1. 2018Generalized Byzantine-tolerant SGDcited by 95
  2. 2018Byzantine Stochastic Gradient Descentcited by 81
  3. 2018Robust Estimation via Robust Gradient Estimationcited by 76
  4. 2021A general sample complexity analysis of vanilla policy gradientcited by 76
  5. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 70
  6. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 68
  7. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
  8. 2022Byzantine Machine Learning Made Easy by Resilient Averaging of Momentumscited by 63
  9. 2019Sample Efficient Policy Gradient Methods with Recursive Variance Reductioncited by 60
  10. 2019An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradientcited by 59
  11. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 58
  12. 2020Byzantine-Resilient Non-Convex Stochastic Gradient Descentcited by 57
  13. 2019DeepPool: Distributed Model-free Algorithm for Ride-sharing using Deep Reinforcement Learningcited by 54
  14. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 54
  15. 2021Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approachcited by 50
  16. 2016Geometric Median in Nearly Linear Timecited by 39
  17. 2022i-Sim2Real: Reinforcement Learning of Robotic Policies in Tight Human-Robot Interaction Loopscited by 32
  18. 2021Byzantine-robust Federated Learning through Collaborative Malicious Gradient Filteringcited by 30
  19. 2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
  20. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 27
  21. 2023Improved Sample Complexity Analysis of Natural Policy Gradient Algorithm with General Parameterization for Infinite Horizon Discounted Reward Markov Decision Processescited by 25
  22. 2023Model-free Learning with Heterogeneous Dynamical Systems: A Federated LQR Approachcited by 23
  23. 2023Improved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updatescited by 21
  24. 2022Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Functioncited by 20
  25. 2022Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Controlcited by 16
  26. 2021Better SGD using Second-order Momentumcited by 15
  27. 2022Byzantine-Robust Online and Offline Distributed Reinforcement Learningcited by 13
  28. 2021Communication Efficient Parallel Reinforcement Learningcited by 10
  29. 2021Decision Making in Monopoly using a Hybrid Deep Reinforcement Learning Approachcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysiscited by 21
  2. 2025Gap-Dependent Bounds for Federated Q-learningcited by 4
  3. 2025Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learningcited by 2
  4. 2025Parameter-Free Federated TD Learning with Markov Noise in Heterogeneous Environmentscited by 1
  5. 2026Provably Efficient Federated Reinforcement Learning with Linear Function Approximation and Logarithmic Communication Costcited by 1
  6. 2026Robust Federated Q-Learning with Almost No Communicationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Decentralized Federated Policy Gradient with Byzantine Fault-Tolerance and Provably Fast Convergence
  2. 2025Provably Robust Federated Reinforcement Learning
  3. 2026Learning from Unreliable Trajectories: Adversarially-Robust Federated Q-Learning
  4. 2025On Global Convergence Rates for Federated Softmax Policy Gradient under Heterogeneous Environments
  5. 2024Towards Fast Rates for Federated and Multi-Task Reinforcement Learning
  6. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments
  7. 2025Federated Reinforcement Learning in Heterogeneous Environments
  8. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning