mlsanityData to 7 Oct 2026

Federated Ensemble-Directed Offline Reinforcement Learning

arXiv 2305.03097 ↗First posted 4 May 2023cs.LG cs.AIcited by 6

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We consider the problem of federated offline reinforcement learning (RL), a scenario under which distributed learning agents must collaboratively learn a high-quality control policy only using small pre-collected datasets generated according to different unknown behavior policies. Na"ively combining a standard offline RL approach with a standard federated learning approach to solve this problem can lead to poorly performing policies. In response, we develop the Federated Ensemble-Directed Offline Reinforcement Learning Algorithm (FEDORA), which distills the collective wisdom of the clients using an ensemble learning approach. We develop the FEDORA codebase to utilize distributed compute resources on a federated learning platform. We show that FEDORA significantly outperforms other approaches, including offline RL over the combined data pool, in various complex continuous control environments and real-world datasets. Finally, we demonstrate the performance of FEDORA in the real-world on a mobile robot. We provide our code and a video of our experiments at urlhttps://github.com/DesikRengarajan/FEDORA.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 44 references that are in the map. First posted on arXiv 4 May 2023.

Builds on

30 of 44 references.

Closest ancestors · 10

Same direction, newest first

  1. 2022Boosting Offline Reinforcement Learning via Data Rebalancing17 Oct
  2. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning16 Sep
  5. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  6. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 20

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2020Adaptive Federated Optimizationcited by 1,036
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  11. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  12. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 449
8 more
  1. 2021A Field Guide to Federated Optimizationcited by 312
  2. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  3. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 79
  4. 2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
  5. 2020CityLearn: Standardizing Research in Multi-Agent Reinforcement Learning for Demand Response and Urban Energy Managementcited by 27
  6. 2022Federated Offline Reinforcement Learningcited by 13
  7. 2020Heterogeneous Data-Aware Federated Learningcited by 9
  8. 2022FedFormer: Contextual Federation with Attention in Reinforcement Learningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 4.

6 papers.

  1. 2025FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Datacited by 5
  2. 2026FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectificationcited by 2
  3. 2024Federated Offline Policy Optimization with Dual Regularizationcited by 1
  4. 2026Federated Distributional Reinforcement Learning with Distributional Critic Regularizationcited by 1
  5. 2026When Backdoors Meet Partial Observability: Attacking Real-World Reinforcement Learningcited by 0
  6. 2026Privacy Preserving Reinforcement Learning with One-Sided Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices
  2. 2022On Decentralizing Federated Reinforcement Learning in Multi-Robot Scenarios
  3. 2026An Empirical Study and Open Testbed for Federated Fine-Tuning of Vision-Language-Action Models
  4. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  6. 2024Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learning
  7. 2026FedMOA: Federated GRPO for Personalized Reasoning LLMs under Heterogeneous Rewards
  8. 2026Trapped by Their Own Rollouts: Understanding Aggregation--Rollout Feedback in Federated On-Policy Distillation