mlsanityData to 14 Oct 2026

Momentum-Based Federated Reinforcement Learning with Interaction and Communication Efficiency

arXiv 2405.17471 ↗First posted 24 May 2024cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → RL for energy, networks and operations

Abstract (arXiv)

Federated Reinforcement Learning (FRL) has garnered increasing attention recently. However, due to the intrinsic spatio-temporal non-stationarity of data distributions, the current approaches typically suffer from high interaction and communication costs. In this paper, we introduce a new FRL algorithm, named textttMFPO, that utilizes momentum, importance sampling, and additional server-side adjustment to control the shift of stochastic policy gradients and enhance the efficiency of data utilization. We prove that by proper selection of momentum parameters and interaction frequency, textttMFPO can achieve tildeO(H N-1ε-3/2) and tildeO(ε-1) interaction and communication complexities (N represents the number of agents), where the interaction complexity achieves linear speedup with the number of agents, and the communication complexity aligns the best achievable of existing first-order FL algorithms. Extensive experiments corroborate the substantial performance gains of textttMFPO over existing methods on a suite of complex and high-dimensional benchmarks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods. Builds on 21 of its 38 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

21 of 38 references.

References · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2016OpenAI Gymcited by 1,736
  6. 2019Challenges of Real-World Reinforcement Learningcited by 291
  7. 2019Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computingcited by 228
  8. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  9. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
  10. 2018Stochastic Variance-Reduced Policy Gradientcited by 106
  11. 2019Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systemscited by 78
  12. 2019An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradientcited by 59
9 more
  1. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 54
  2. 2019Federated Transfer Reinforcement Learning for Autonomous Drivingcited by 44
  3. 2021STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learningcited by 43
  4. 2019Federated Deep Reinforcement Learningcited by 35
  5. 2019The Complexity of Finding Stationary Points with Stochastic Gradient Descentcited by 35
  6. 2020When Deep Reinforcement Learning Meets Federated Learning: Intelligent Multi-Timescale Resource Management for Multi-access Edge Computing in 5G Ultra Dense Networkcited by 30
  7. 2022Federated Stochastic Approximation under Markov Noise and Heterogeneity: Applications in Reinforcement Learningcited by 27
  8. 2021Multi-Task Federated Reinforcement Learning with Adversariescited by 13
  9. 2019Federated Reinforcement Distillation with Proxy Experience Memorycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments
  2. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learning
  3. 2024Towards Fast Rates for Federated and Multi-Task Reinforcement Learning
  4. 2026Provably Efficient Federated Reinforcement Learning with Linear Function Approximation and Logarithmic Communication Cost
  5. 2025Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Sampling
  6. 2025Parameter-Free Federated TD Learning with Markov Noise in Heterogeneous Environments
  7. 2026Personalized Federated Reinforcement Learning via Model-Agnostic Meta-Learning: Convergence of Exact and Hessian-Free Meta-Policy Gradients
  8. 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis