mlsanityData to 14 Oct 2026

Improved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updates

arXiv 2310.19807 ↗First posted 9 Oct 2023cs.LG math.OCcited by 21

Not on the map. Why?

Abstract (arXiv)

Federated reinforcement learning (FedRL) enables agents to collaboratively train a global policy without sharing their individual data. However, high communication overhead remains a critical bottleneck, particularly for natural policy gradient (NPG) methods, which are second-order. To address this issue, we propose the FedNPG-ADMM framework, which leverages the alternating direction method of multipliers (ADMM) to approximate global NPG directions efficiently. We theoretically demonstrate that using ADMM-based gradient updates reduces communication complexity from O(d2) to O(d) at each iteration, where d is the number of model parameters. Furthermore, we show that achieving an ε-error stationary convergence requires O(frac1(1-γ)2ε) iterations for discount factor γ, demonstrating that FedNPG-ADMM maintains the same convergence rate as the standard FedNPG. Through evaluation of the proposed algorithms in MuJoCo environments, we demonstrate that FedNPG-ADMM maintains the reward performance of standard FedNPG, and that its convergence rate improves when the number of federated agents increases.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

36 of 51 references.

References · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  7. 2019Advances and Open Problems in Federated Learningcited by 3,046
  8. 2015Trust Region Policy Optimizationcited by 2,580
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  10. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
  11. 2021Federated Learning for Internet of Things: A Comprehensive Surveycited by 252
  12. 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
24 more
  1. 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 132
  2. 2019Federated Learning for Wireless Communications: Motivation, Opportunities and Challengescited by 129
  3. 2018Stochastic Variance-Reduced Policy Gradientcited by 106
  4. 2017Towards Generalization and Simplicity in Continuous Controlcited by 68
  5. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 68
  6. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
  7. 2019Sample Efficient Policy Gradient Methods with Recursive Variance Reductioncited by 60
  8. 2019DeepPool: Distributed Model-free Algorithm for Ride-sharing using Deep Reinforcement Learningcited by 54
  9. 2021FedNL: Making Newton-Type Methods Applicable to Federated Learningcited by 52
  10. 2022FedADMM: A Federated Primal-Dual Algorithm Allowing Partial Participationcited by 36
  11. 2019GADMM: Fast and Communication Efficient Framework for Distributed Machine Learningcited by 30
  12. 2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
  13. 2022Federated Stochastic Approximation under Markov Noise and Heterogeneity: Applications in Reinforcement Learningcited by 27
  14. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 27
  15. 2021Distributed Second Order Methods with Fast Rates and Compressed Communicationcited by 25
  16. 2017Bandit Convex Optimization for Scalable and Dynamic IoT Managementcited by 24
  17. 2022FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type Method for Federated Learningcited by 23
  18. 2019Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learningcited by 17
  19. 2020A Distributed Model-Free Ride-Sharing Approach for Joint Matching, Pricing, and Dispatching using Deep Reinforcement Learningcited by 17
  20. 2022FedSysID: A Federated Approach to Sample-Efficient System Identificationcited by 17
  21. 2016Convergence Rate of Distributed ADMM over Networkscited by 16
  22. 2021Basis Matters: Better Communication-Efficient Second Order Methods for Federated Learningcited by 15
  23. 2021Communication Efficient Parallel Reinforcement Learningcited by 10
  24. 2021ElegantRL-Podracer: Scalable and Elastic Library for Cloud-Native Deep Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 5; 2026: 1.

21 papers.

Sort:

By area:

  1. 2024Credit Card Fraud Detection Using Advanced Transformer Modelcited by 26
  2. 2024Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detectioncited by 22
  3. 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysiscited by 21
  4. 2024Real-Time Pill Identification for the Visually Impaired Using Deep Learningcited by 21
  5. 2024Efficiency optimization of large-scale language models based on deep learning in natural language processing taskscited by 14
  6. 2024Exploration of Multi-Scale Image Fusion Systems in Intelligent Medical Image Analysiscited by 10
  7. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environmentscited by 8
  8. 2024Advanced Payment Security System:XGBoost, LightGBM and SMOTE Integratedcited by 8
  9. 2024Advancements in Feature Extraction Recognition of Medical Imaging Systems Through Deep Learning Techniquecited by 7
  10. 2024In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before An Ongoing Trajectory Terminatescited by 7
  11. 2025One-Shot Federated Learning with Classifier-Free Diffusion Modelscited by 7
  12. 2024Global Convergence Guarantees for Federated Policy Gradient Methods with Adversariescited by 6
9 more
  1. 2024Advanced Multimodal Deep Learning Architecture for Image-Text Matchingcited by 4
  2. 2024Towards Fast Rates for Federated and Multi-Task Reinforcement Learningcited by 4
  3. 2025Gap-Dependent Bounds for Federated Q-learningcited by 4
  4. 2025Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Samplingcited by 2
  5. 2025Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learningcited by 2
  6. 2025Convergence Guarantees for Federated SARSA with Local Training and Heterogeneous Agentscited by 2
  7. 2023Communication Efficient and Privacy-Preserving Federated Learning Based on Evolution Strategiescited by 0
  8. 2024StraightLine: An End-to-End Resource-Aware Scheduler for Machine Learning Application Requestscited by 0
  9. 2026Learning from Unreliable Trajectories: Adversarially-Robust Federated Q-Learningcited by 0