mlsanityData to 7 Oct 2026

Federated Reinforcement Learning: Techniques, Applications, and Open Challenges

arXiv 2108.11887 ↗First posted 26 Aug 2021cs.LG cs.AIcited by 79

Not on the map. Why?

Abstract (arXiv)

This paper presents a comprehensive survey of Federated Reinforcement Learning (FRL), an emerging and promising field in Reinforcement Learning (RL). Starting with a tutorial of Federated Learning (FL) and RL, we then focus on the introduction of FRL as a new method with great potential by leveraging the basic idea of FL to improve the performance of RL while preserving data-privacy. According to the distribution characteristics of the agents in the framework, FRL algorithms can be divided into two categories, i.e. Horizontal Federated Reinforcement Learning (HFRL) and Vertical Federated Reinforcement Learning (VFRL). We provide the detailed definitions of each category by formulas, investigate the evolution of FRL from a technical perspective, and highlight its advantages over previous RL algorithms. In addition, the existing works on FRL are summarized by application fields, including edge computing, communication, control optimization, and attack detection. Finally, we describe and discuss several key research directions that are crucial to solving the open problems within FRL.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Builds on 68 of its 135 references that are in the map. First posted on arXiv 26 Aug 2021.

Builds on

68 of 135 references.

References · 68

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  9. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  10. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  11. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  12. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
38 more
  1. 2015Prioritized Experience Replaycited by 1,070
  2. 2019Deep Leakage from Gradientscited by 1,004
  3. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  4. 2019Towards Federated Learning at Scale: System Designcited by 826
  5. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  6. 2017Deep Reinforcement Learning that Matterscited by 727
  7. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  8. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  9. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  10. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 490
  11. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  12. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 408
  13. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 407
  14. 2016Learning Multiagent Communication with Backpropagationcited by 355
  15. 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 347
  16. 2017Deep Reinforcement Learning: An Overviewcited by 327
  17. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
  18. 2018Distributed Prioritized Experience Replaycited by 260
  19. 2021Federated Learning for Internet of Things: A Comprehensive Surveycited by 247
  20. 2013The Complexity of Decentralized Control of Markov Decision Processescited by 225
  21. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  22. 2017Multi-agent Reinforcement Learning in Sequential Social Dilemmascited by 206
  23. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 202
  24. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  25. 2018Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agentscited by 189
  26. 2015Multiagent Cooperation and Competition with Deep Reinforcement Learningcited by 180
  27. 2020Threats to Federated Learning: A Surveycited by 171
  28. 2018Mitigating Sybils in Federated Learning Poisoningcited by 158
  29. 2019A Review of Cooperative Multi-Agent Deep Reinforcement Learningcited by 147
  30. 2017Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observabilitycited by 143
  31. 2017Revisiting Distributed Synchronous SGDcited by 142
  32. 2020Federated Learning for Internet of Things: Recent Advances, Taxonomy, and Open Challengescited by 138
  33. 2017Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learningcited by 114
  34. 2020Convergence Time Optimization for Federated Learning over Wireless Networkscited by 81
  35. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 80
  36. 2018In-Edge AI: Intelligentizing Mobile Edge Computing, Caching and Communication by Federated Learningcited by 80
  37. 2019Measure Contribution of Participants in Federated Learningcited by 78
  38. 2015Memory-based control with recurrent neural networkscited by 77

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 12; 2023: 21; 2024: 14; 2025: 16; 2026: 13.

79 papers.

Sort:

By area:

  1. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 38
  2. 2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
  3. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learningcited by 29
  4. 2023Federated TD Learning over Finite-Rate Erasure Channels: Linear Speedup under Markovian Samplingcited by 14
  5. 2022Autonomy and Intelligence in the Computing Continuum: Challenges, Enablers, and Future Directions for Orchestrationcited by 12
  6. 2022Federated Learning and Meta Learning: Approaches, Applications, and Directionscited by 12
  7. 2023Temporal Difference Learning with Compressed Updates: Error-Feedback meets Reinforcement Learningcited by 12
  8. 2023Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learningcited by 11
  9. 2024Training Machine Learning models at the Edge: A Surveycited by 11
  10. 2023Sharing Lifelong Reinforcement Learning Knowledge via Modulating Maskscited by 10
  11. 2023An Analysis of Multi-Agent Reinforcement Learning for Decentralized Inventory Control Systemscited by 10
  12. 2022Linear Stochastic Bandits over a Bit-Constrained Channelcited by 9
38 more
  1. 2023Cooperative Multi-Agent Reinforcement Learning: Asynchronous Communication and Linear Function Approximationcited by 9
  2. 2022FedFormer: Contextual Federation with Attention in Reinforcement Learningcited by 8
  3. 2023Fair yet Asymptotically Equal Collaborative Learningcited by 8
  4. 2023Advancements in Federated Learning: Models, Methods, and Privacycited by 7
  5. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environmentscited by 7
  6. 2021Semantic-Aware Collaborative Deep Reinforcement Learning Over Wireless Cellular Networkscited by 6
  7. 2023Federated Ensemble-Directed Offline Reinforcement Learningcited by 6
  8. 2024SCAFFLSA: Taming Heterogeneity in Federated Linear Stochastic Approximation and TD Learningcited by 6
  9. 2025DeepSeek-Inspired Exploration of RL-based LLMs and Synergy with Wireless Networks: A Surveycited by 6
  10. 2024Federated UCBVI: Communication-Efficient Federated Regret Minimization with Heterogeneous Agentscited by 5
  11. 2025FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Datacited by 5
  12. 2022Federated Reinforcement Learning for Real-Time Electric Vehicle Charging and Discharging Controlcited by 4
  13. 2022Differentiated Federated Reinforcement Learning Based Traffic Offloading on Space-Air-Ground Integrated Networkscited by 4
  14. 2024On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representationscited by 4
  15. 2022Just Round: Quantized Observation Spaces Enable Memory Efficient Learning of Dynamic Locomotioncited by 3
  16. 2023Learning Complex Teamwork Tasks Using a Given Sub-task Decompositioncited by 3
  17. 2023Online Learning with Adversaries: A Differential-Inclusion Analysiscited by 3
  18. 2023Client Selection for Federated Policy Optimization with Environment Heterogeneitycited by 3
  19. 2023Model-aided Federated Reinforcement Learning for Multi-UAV Trajectory Planning in IoT Networkscited by 3
  20. 2023Federated Reinforcement Learning for Resource Allocation in V2X Networkscited by 3
  21. 2024Towards Fast Rates for Federated and Multi-Task Reinforcement Learningcited by 3
  22. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Reviewcited by 3
  23. 2025Heterogeneous Federated Reinforcement Learning Using Wasserstein Barycenterscited by 3
  24. 2022Decentralized Federated Reinforcement Learning for User-Centric Dynamic TFDD Controlcited by 2
  25. 2023Federated Reinforcement Learning for Electric Vehicles Charging Control on Distribution Networkscited by 2
  26. 2024Decentralized Federated Policy Gradient with Byzantine Fault-Tolerance and Provably Fast Convergencecited by 2
  27. 2024CAESAR: Enhancing Federated RL in Heterogeneous MDPs through Convergence-Aware Sampling with Screeningcited by 2
  28. 2024Combining Federated Learning and Control: A Surveycited by 2
  29. 2024On the Convergence Rates of Federated Q-Learning across Heterogeneous Environmentscited by 2
  30. 2025Collaborative Value Function Estimation Under Model Mismatch: A Federated Temporal Difference Analysiscited by 2
  31. 2025Convergence Guarantees for Federated SARSA with Local Training and Heterogeneous Agentscited by 2
  32. 2026FedMOA: Federated GRPO for Personalized Reasoning LLMs under Heterogeneous Rewardscited by 2
  33. 2026FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectificationcited by 2
  34. 2022Personalized Federated Hypernetworks for Privacy Preservation in Multi-Task Reinforcement Learningcited by 1
  35. 2022Enhancing Cyber Resilience of Networked Microgrids using Vertical Federated Reinforcement Learningcited by 1
  36. 2023Generalization of Deep Reinforcement Learning for Jammer-Resilient Frequency and Power Allocationcited by 1
  37. 2023A Federated Reinforcement Learning Framework for Link Activation in Multi-link Wi-Fi Networkscited by 1
  38. 2024SwiftRL: Towards Efficient Reinforcement Learning on Real Processing-In-Memory Systemscited by 1

50 most cited of 79 All 79 (JSON)