mlsanityData to 7 Oct 2026

Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environments

arXiv 2405.19499 ↗First posted 29 May 2024cs.LG cs.MA math.OCcited by 7

Not on the map. Why?

Abstract (arXiv)

We explore a Federated Reinforcement Learning (FRL) problem where N agents collaboratively learn a common policy without sharing their trajectory data. To date, existing FRL work has primarily focused on agents operating in the same or ``similar" environments. In contrast, our problem setup allows for arbitrarily large levels of environment heterogeneity. To obtain the optimal policy which maximizes the average performance across all potentially completely different environments, we propose two algorithms: FedSVRPG-M and FedHAPG-M. In contrast to existing results, we demonstrate that both FedSVRPG-M and FedHAPG-M, both of which leverage momentum mechanisms, can exactly converge to a stationary point of the average performance function, regardless of the magnitude of environment heterogeneity. Furthermore, by incorporating the benefits of variance-reduction techniques or Hessian approximation, both algorithms achieve state-of-the-art convergence results, characterized by a sample complexity of Oleft(ε^-frac32/Nright). Notably, our algorithms enjoy linear convergence speedups with respect to the number of agents, highlighting the benefit of collaboration among agents in finding a common policy.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 35 of its 45 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

35 of 45 references.

References · 35

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  5. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  6. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  7. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  8. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  9. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  10. 2018Stochastic Variance-Reduced Policy Gradientcited by 103
  11. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 100
  12. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 79
23 more
  1. 2019Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systemscited by 77
  2. 2022An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methodscited by 63
  3. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 61
  4. 2019An Improved Convergence Analysis of Stochastic Variance-Reduced Policy Gradientcited by 58
  5. 2019Sample Efficient Policy Gradient Methods with Recursive Variance Reductioncited by 58
  6. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 53
  7. 2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
  8. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 52
  9. 2023Momentum Benefits Non-IID Federated Learning Simply and Provablycited by 37
  10. 2022FedADMM: A Federated Primal-Dual Algorithm Allowing Partial Participationcited by 35
  11. 2023The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyondcited by 34
  12. 2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
  13. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learningcited by 29
  14. 2020Momentum-Based Policy Gradient Methodscited by 28
  15. 2022Federated Stochastic Approximation under Markov Noise and Heterogeneity: Applications in Reinforcement Learningcited by 27
  16. 2021Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularizationcited by 25
  17. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 25
  18. 2020A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learningcited by 24
  19. 2020Stochastic Recursive Momentum for Policy Gradient Methodscited by 22
  20. 2023Improved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updatescited by 20
  21. 2023Model-free Learning with Heterogeneous Dynamical Systems: A Federated LQR Approachcited by 19
  22. 2022PAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method with Probabilistic Gradient Estimationcited by 15
  23. 2023Client Selection for Federated Policy Optimization with Environment Heterogeneitycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2025FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Datacited by 5
  2. 2024Towards Fast Rates for Federated and Multi-Task Reinforcement Learningcited by 3
  3. 2025Convergence Guarantees for Federated SARSA with Local Training and Heterogeneous Agentscited by 2
  4. 2025Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Samplingcited by 1
  5. 2026Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learningcited by 0
  6. 2026FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learningcited by 0
  7. 2026When Do Causal World Models Help Modular LLM Agentscited by 0