mlsanityData to 7 Oct 2026

Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis

arXiv 2404.08003 ↗First posted 9 Apr 2024cs.LG cs.DC cs.NIcited by 21

Not on the map. Why?

Abstract (arXiv)

To improve the efficiency of reinforcement learning (RL), we propose a novel asynchronous federated reinforcement learning (FedRL) framework termed AFedPG, which constructs a global model through collaboration among N agents using policy gradient (PG) updates. To address the challenge of lagged policies in asynchronous settings, we design a delay-adaptive lookahead technique textitspecifically for FedRL that can effectively handle heterogeneous arrival times of policy gradients. We analyze the theoretical global convergence bound of AFedPG, and characterize the advantage of the proposed algorithm in terms of both the sample complexity and time complexity. Specifically, our AFedPG method achieves O(fracε-2.5N) sample complexity for global convergence at each agent on average. Compared to the single agent setting with O(ε-2.5) sample complexity, it enjoys a linear speedup with respect to the number of agents. Moreover, compared to synchronous FedPG, AFedPG improves the time complexity from O(fractmaxN) to O(sumi=1N frac1ti)-1, where ti denotes the time consumption in each iteration at agent i, and tmax is the largest one. The latter complexity O(sumi=1N frac1ti)-1 is always smaller than the former one, and this improvement becomes significant in large-scale federated settings with heterogeneous computing powers (tmaxgg tmin). Finally, we empirically verify the improved performance of AFedPG in four widely used MuJoCo environments with varying numbers of agents. We also demonstrate the advantages of AFedPG in various computing heterogeneity scenarios.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 47 of its 77 references that are in the map. First posted on arXiv 9 Apr 2024.

Builds on

47 of 77 references.

References · 47

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  9. 2019Asynchronous Federated Optimizationcited by 264
  10. 2021Federated Learning for Internet of Things: A Comprehensive Surveycited by 247
  11. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
  12. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
35 more
  1. 2019Federated Learning for Wireless Communications: Motivation, Opportunities and Challengescited by 128
  2. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 117
  3. 2018Stochastic Variance-Reduced Policy Gradientcited by 103
  4. 2020VAFL: a Method of Vertical Asynchronous Federated Learningcited by 87
  5. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 76
  6. 2021A general sample complexity analysis of vanilla policy gradientcited by 74
  7. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 63
  8. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 61
  9. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 61
  10. 2019Sample Efficient Policy Gradient Methods with Recursive Variance Reductioncited by 58
  11. 2019DeepPool: Distributed Model-free Algorithm for Ride-sharing using Deep Reinforcement Learningcited by 54
  12. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 53
  13. 2023The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyondcited by 34
  14. 2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
  15. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learningcited by 29
  16. 2020Momentum-Based Policy Gradient Methodscited by 28
  17. 2022Federated Stochastic Approximation under Markov Noise and Heterogeneity: Applications in Reinforcement Learningcited by 27
  18. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 25
  19. 2023Improved Sample Complexity Analysis of Natural Policy Gradient Algorithm with General Parameterization for Infinite Horizon Discounted Reward Markov Decision Processescited by 24
  20. 2020Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedupcited by 23
  21. 2021On the Global Optimum Convergence of Momentum-based Policy Gradientcited by 23
  22. 2023Improved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updatescited by 20
  23. 2019Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learningcited by 17
  24. 2020A Distributed Model-Free Ride-Sharing Approach for Joint Matching, Pricing, and Dispatching using Deep Reinforcement Learningcited by 17
  25. 2022Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropoutcited by 12
  26. 2023Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learningcited by 11
  27. 2021Communication Efficient Parallel Reinforcement Learningcited by 10
  28. 2021ElegantRL-Podracer: Scalable and Elastic Library for Cloud-Native Deep Reinforcement Learningcited by 9
  29. 2022Asynchronous Hybrid Reinforcement Learning for Latency and Reliability Optimization in the Metaverse over Wireless Communicationscited by 9
  30. 2023Sample Efficient Reinforcement Learning in Mixed Systems through Augmented Samples and Its Applications to Queueing Networkscited by 9
  31. 2024Global Convergence Guarantees for Federated Policy Gradient Methods with Adversariescited by 6
  32. 2023Learning-based Two-tiered Online Optimization of Region-wide Datacenter Resource Allocationcited by 4
  33. 2022Effective Multi-User Delay-Constrained Scheduling with Deep Recurrent Reinforcement Learningcited by 3
  34. 2020Cross Layer Optimization and Distributed Reinforcement Learning for Wireless 360deg Video Streamingcited by 1
  35. 2023Federated Learning over Hierarchical Wireless Networks: Training Latency Minimization via Submodel Partitioningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 6; 2026: 5.

21 papers.

Sort:

By area:

  1. 2024Credit Card Fraud Detection Using Advanced Transformer Modelcited by 26
  2. 2024Real-Time Pill Identification for the Visually Impaired Using Deep Learningcited by 20
  3. 2024Efficiency optimization of large-scale language models based on deep learning in natural language processing taskscited by 14
  4. 2025A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximationcited by 11
  5. 2024Exploration of Multi-Scale Image Fusion Systems in Intelligent Medical Image Analysiscited by 10
  6. 2024The Sample-Communication Complexity Trade-off in Federated Q-Learningcited by 8
  7. 2024Advancements in Feature Extraction Recognition of Medical Imaging Systems Through Deep Learning Techniquecited by 7
  8. 2024In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before An Ongoing Trajectory Terminatescited by 7
  9. 2024Advanced Multimodal Deep Learning Architecture for Image-Text Matchingcited by 4
  10. 2024On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representationscited by 4
  11. 2025Gap-Dependent Bounds for Federated Q-learningcited by 4
  12. 2025Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learningcited by 2
9 more
  1. 2025Convergence Guarantees for Federated SARSA with Local Training and Heterogeneous Agentscited by 2
  2. 2025Nonparametric Bellman Mappings for Value Iteration in Distributed Reinforcement Learningcited by 1
  3. 2025Asynchronous Policy Gradient Aggregation for Efficient Distributed Reinforcement Learningcited by 1
  4. 2024StraightLine: An End-to-End Resource-Aware Scheduler for Machine Learning Application Requestscited by 0
  5. 2026FedPSA: Modeling Behavioral Staleness in Asynchronous Federated Learningcited by 0
  6. 2026UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Modelscited by 0
  7. 2026Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learningcited by 0
  8. 2026Provably Efficient Federated Reinforcement Learning with Linear Function Approximation and Logarithmic Communication Costcited by 0
  9. 2026Personalized Federated Reinforcement Learning via Model-Agnostic Meta-Learning: Convergence of Exact and Hessian-Free Meta-Policy Gradientscited by 0