mlsanityData to 14 Oct 2026

FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergence

arXiv 2204.08125 ↗First posted 18 Apr 2022cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

As a distributed learning paradigm, Federated Learning (FL) faces the communication bottleneck issue due to many rounds of model synchronization and aggregation. Heterogeneous data further deteriorates the situation by causing slow convergence. Although the impact of data heterogeneity on supervised FL has been widely studied, the related investigation for Federated Reinforcement Learning (FRL) is still in its infancy. In this paper, we first define the type and level of data heterogeneity for policy gradient based FRL systems. By inspecting the connection between the global and local objective functions, we prove that local training can benefit the global objective, if the local update is properly penalized by the total variation (TV) distance between the local and global policies. A necessary condition for the global policy to be learn-able from the local policy is also derived, which is directly related to the heterogeneity level. Based on the theoretical result, a Kullback-Leibler (KL) divergence based penalty is proposed, which, different from the conventional method that penalizes the model divergence in the parameter space, directly constrains the model outputs in the distribution space. Convergence proof of the proposed algorithm is also provided. By jointly penalizing the divergence of the local policy from the global policy with a global penalty and constraining each iteration of the local training with a local penalty, the proposed method achieves a better trade-off between training speed (step size) and convergence. Experiment results on two popular Reinforcement Learning (RL) experiment platforms demonstrate the advantage of the proposed algorithm over existing methods in accelerating and stabilizing the training process with heterogeneous data.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 28 of its 48 references that are in the map. First posted on arXiv 18 Apr 2022.

Builds on

28 of 48 references.

References · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2016OpenAI Gymcited by 1,736
  7. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  8. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  9. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  10. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  11. 2018Federated Learning with Non-IID Datacited by 1,108
  12. 2017Constrained Policy Optimizationcited by 983
16 more
  1. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  2. 2019Federated Machine Learning: Concept and Applicationscited by 412
  3. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  4. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  5. 2019Client-Edge-Cloud Hierarchical Federated Learningcited by 202
  6. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 186
  7. 2016QSGD: Communication-Efficient SGD via Gradient Quantization and Encodingcited by 147
  8. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 107
  9. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 81
  10. 2019Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systemscited by 78
  11. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
  12. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 54
  13. 2019Federated Transfer Reinforcement Learning for Autonomous Drivingcited by 44
  14. 2018Expected Policy Gradients for Reinforcement Learningcited by 28
  15. 2018Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learningcited by 11
  16. 2021The Gradient Convergence Bound of Federated Multi-Agent Reinforcement Learning with Efficient Communicationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 6; 2024: 13; 2025: 6; 2026: 2.

29 papers.

Sort:

By area:

  1. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learningcited by 31
  2. 2023Improved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updatescited by 21
  3. 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysiscited by 21
  4. 2022Federated Offline Reinforcement Learningcited by 13
  5. 2022Stateful active facilitator: Coordination and Environmental Heterogeneity in Cooperative Multi-Agent Reinforcement Learningcited by 10
  6. 2023Federated Learning Across Decentralized and Unshared Archives for Remote Sensing Image Classificationcited by 10
  7. 2024The Sample-Communication Complexity Trade-off in Federated Q-Learningcited by 9
  8. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environmentscited by 8
  9. 2024SCAFFLSA: Taming Heterogeneity in Federated Linear Stochastic Approximation and TD Learningcited by 7
  10. 2023Federated Ensemble-Directed Offline Reinforcement Learningcited by 6
  11. 2024Global Convergence Guarantees for Federated Policy Gradient Methods with Adversariescited by 6
  12. 2025FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Datacited by 5
17 more
  1. 2024Federated Learning Resilient to Byzantine Attacks and Data Heterogeneitycited by 4
  2. 2024Towards Fast Rates for Federated and Multi-Task Reinforcement Learningcited by 4
  3. 2024FedCFA: Alleviating Simpson's Paradox in Model Aggregation with Counterfactual Federated Learningcited by 4
  4. 2023Client Selection for Federated Policy Optimization with Environment Heterogeneitycited by 3
  5. 2023Analysis and Optimization of Wireless Federated Learning with Data Heterogeneitycited by 3
  6. 2024Energy-Efficient Channel Decoding for Wireless Federated Learning: Convergence Analysis and Adaptive Designcited by 3
  7. 2024Efficient Federated Learning against Byzantine Attacks and Data Heterogeneity via Aggregating Normalized Gradientscited by 2
  8. 2024On the Convergence Rates of Federated Q-Learning across Heterogeneous Environmentscited by 2
  9. 2025Collaborative Value Function Estimation Under Model Mismatch: A Federated Temporal Difference Analysiscited by 2
  10. 2025Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Samplingcited by 2
  11. 2025The Actor-Critic Update Order Matters for PPO in Federated Reinforcement Learningcited by 2
  12. 2025Federated Hierarchical Reinforcement Learning for Adaptive Traffic Signal Controlcited by 1
  13. 2023Epidemic Decision-making System Based Federated Reinforcement Learningcited by 0
  14. 2024Single-Loop Federated Actor-Critic across Heterogeneous Environmentscited by 0
  15. 2025Federated Reinforcement Learning in Heterogeneous Environmentscited by 0
  16. 2026On the Effectiveness of Adaptation Strategies for VLM-Based Federated Learning in Remote Sensingcited by 0
  17. 2026FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learningcited by 0