mlsanityData to 7 Oct 2026

Federated Reinforcement Learning with Environment Heterogeneity

arXiv 2204.02634 ↗First posted 6 Apr 2022cs.LG stat.MLcited by 61

Not on the map. Why?

Abstract (arXiv)

We study a Federated Reinforcement Learning (FedRL) problem in which n agents collaboratively learn a single policy without sharing the trajectories they collected during agent-environment interaction. We stress the constraint of environment heterogeneity, which means n environments corresponding to these n agents have different state transitions. To obtain a value function or a policy function which optimizes the overall performance in all environments, we propose two federated RL algorithms, textttQAvg and textttPAvg. We theoretically prove that these algorithms converge to suboptimal solutions, while such suboptimality depends on how heterogeneous these n environments are. Moreover, we propose a heuristic that achieves personalization by embedding the n environments into n vectors. The personalization heuristic not only improves the training but also allows for better generalization to new environments.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 16 of its 29 references that are in the map. First posted on arXiv 6 Apr 2022.

Builds on

16 of 29 references.

References · 16

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  4. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  5. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  7. 2019Federated Learning with Personalization Layerscited by 533
  8. 2020Adaptive Personalized Federated Learningcited by 353
  9. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
  10. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  11. 2019Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systemscited by 77
  12. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 68
4 more
  1. 2019Federated User Representation Learningcited by 31
  2. 2017Robust and Efficient Transfer Learning with Hidden-Parameter Markov Decision Processescited by 28
  3. 2020A Decentralized Policy Gradient Approach to Multi-task Reinforcement Learningcited by 24
  4. 2018Fingerprint Policy Optimisation for Robust Reinforcement Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 11; 2024: 19; 2025: 16; 2026: 12.

61 papers.

Sort:

By area:

  1. 2023The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyondcited by 34
  2. 2023Momentum Provably Improves Error Feedback!cited by 31
  3. 2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
  4. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learningcited by 29
  5. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 25
  6. 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysiscited by 21
  7. 2023Improved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updatescited by 20
  8. 2023Temporal Difference Learning with Compressed Updates: Error-Feedback meets Reinforcement Learningcited by 12
  9. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Sufficescited by 11
  10. 2022Stateful active facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learningcited by 10
  11. 2023Cooperative Multi-Agent Reinforcement Learning: Asynchronous Communication and Linear Function Approximationcited by 9
  12. 2024Non-convex Stochastic Composite Optimization with Polyak Momentumcited by 9
38 more
  1. 2024The Sample-Communication Complexity Trade-off in Federated Q-Learningcited by 8
  2. 2025Provably Robust Federated Reinforcement Learningcited by 8
  3. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environmentscited by 7
  4. 2023FedHQL: Federated Heterogeneous Q-Learningcited by 6
  5. 2024SCAFFLSA: Taming Heterogeneity in Federated Linear Stochastic Approximation and TD Learningcited by 6
  6. 2024Global Convergence Guarantees for Federated Policy Gradient Methods with Adversariescited by 6
  7. 2024Compressed Federated Reinforcement Learning with a Generative Modelcited by 5
  8. 2024Federated UCBVI: Communication-Efficient Federated Regret Minimization with Heterogeneous Agentscited by 5
  9. 2025FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Datacited by 5
  10. 2022Federated Reinforcement Learning for Real-Time Electric Vehicle Charging and Discharging Controlcited by 4
  11. 2024Towards Faster Decentralized Stochastic Optimization with Communication Compressioncited by 4
  12. 2024On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representationscited by 4
  13. 2025Gap-Dependent Bounds for Federated Q-learningcited by 4
  14. 2023Client Selection for Federated Policy Optimization with Environment Heterogeneitycited by 3
  15. 2023Provably Efficient Offline Reinforcement Learning with Perturbed Data Sourcescited by 3
  16. 2024Towards Fast Rates for Federated and Multi-Task Reinforcement Learningcited by 3
  17. 2025Heterogeneous Federated Reinforcement Learning Using Wasserstein Barycenterscited by 3
  18. 2025Fed-SE: Federated Self-Evolution for Privacy-Constrained Multi-Environment LLM Agentscited by 3
  19. 2024Federated Prompt-based Decision Transformer for Customized VR Services in Mobile Edge Computing Systemcited by 2
  20. 2024Federated Reinforcement Learning with Constraint Heterogeneitycited by 2
  21. 2024On the Convergence Rates of Federated Q-Learning across Heterogeneous Environmentscited by 2
  22. 2025Collaborative Value Function Estimation Under Model Mismatch: A Federated Temporal Difference Analysiscited by 2
  23. 2025Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learningcited by 2
  24. 2025Regret-Optimal Q-Learning with Low Cost for Single-Agent and Federated Reinforcement Learningcited by 2
  25. 2025Convergence Guarantees for Federated SARSA with Local Training and Heterogeneous Agentscited by 2
  26. 2026FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectificationcited by 2
  27. 2023Harnessing the Power of Federated Learning in Federated Contextual Banditscited by 1
  28. 2025Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Samplingcited by 1
  29. 2025The Actor-Critic Update Order Matters for PPO in Federated Reinforcement Learningcited by 1
  30. 2025Asynchronous Policy Gradient Aggregation for Efficient Distributed Reinforcement Learningcited by 1
  31. 2026Federated Distributional Reinforcement Learning with Distributional Critic Regularizationcited by 1
  32. 2023Epidemic Decision-making System Based Federated Reinforcement Learningcited by 0
  33. 2024Federated Control in Markov Decision Processescited by 0
  34. 2024When Swarm Learning meets energy series data: A decentralized collaborative learning design based on blockchaincited by 0
  35. 2024Single-Loop Federated Actor-Critic across Heterogeneous Environmentscited by 0
  36. 2025Approximated Behavioral Metric-based State Projection for Federated Reinforcement Learningcited by 0
  37. 2025Causal Policy Learning in Reinforcement Learning: Backdoor-Adjusted Soft Actor-Criticcited by 0
  38. 2025Federated Reinforcement Learning in Heterogeneous Environmentscited by 0

50 most cited of 61 All 61 (JSON)