Federated Reinforcement Learning in Heterogeneous Environments

arXiv 2507.14487 ↗First posted 19 Jul 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL for energy, networks and operations

Abstract (arXiv)

We investigate a Federated Reinforcement Learning with Environment Heterogeneity (FRL-EH) framework, where local environments exhibit statistical heterogeneity. Within this framework, agents collaboratively learn a global policy by aggregating their collective experiences while preserving the privacy of their local trajectories. To better reflect real-world scenarios, we introduce a robust FRL-EH framework by presenting a novel global objective function. This function is specifically designed to optimize a global policy that ensures robust performance across heterogeneous local environments and their plausible perturbations. We propose a tabular FRL algorithm named FedRQ and theoretically prove its asymptotic convergence to an optimal policy for the global objective function. Furthermore, we extend FedRQ to environments with continuous state space through the use of expectile loss, addressing the key challenge of minimizing a value function over a continuous subset of the state space. This advancement facilitates the seamless integration of the principles of FedRQ with various Deep Neural Network (DNN)-based RL algorithms. Extensive empirical evaluations validate the effectiveness and robustness of our FRL algorithms across diverse heterogeneous environments, consistently achieving superior performance over the existing state-of-the-art FRL algorithms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods. Builds on 21 of its 44 references that are in the map. First posted on arXiv 19 Jul 2025.

Builds on

21 of 44 references.

References · 21

2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
2015Continuous control with deep reinforcement learningcited by 3,407
2013Playing Atari with Deep Reinforcement Learningcited by 3,274
2018Federated Optimization in Heterogeneous Networkscited by 2,987
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
2016OpenAI Gymcited by 1,736
2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
2017A Brief Survey of Deep Reinforcement Learningcited by 483
2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 374
2018Applications of Deep Reinforcement Learning in Communications and Networking: A Surveycited by 167
2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114

9 more

2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 81
2019Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systemscited by 78
2022Tackling Data Heterogeneity in Federated Learning with Class Prototypescited by 67
2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
2019Federated Deep Reinforcement Learningcited by 35
2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
2021Communication-Efficient Online Federated Learning Framework for Nonlinear Regressioncited by 16
2022Tighter Regret Analysis and Optimization of Online Federated Learningcited by 14
2023A Communication-Efficient Adaptive Algorithm for Federated Learning under Cumulative Regretcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way