mlsanityData to 13 Oct 2026

FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Data

arXiv 2512.02350 ↗First posted 2 Dec 2025cs.LG cs.AIcited by 5

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Federated Reinforcement Learning (FRL), a marriage of federated learning and offline reinforcement learning, has attracted increasing interest recently. Albeit with some advancement, we find that the performance of most existing offline FRL methods drops dramatically when provided with mixed-quality data, that is, the logging behaviors (offline data) are collected by policies with varying qualities across clients. To overcome this limitation, this paper introduces a new vote-based offline FRL framework, named FOVA. It exploits a vote mechanism to identify high-return actions during local policy evaluation, alleviating the negative effect of low-quality behaviors from diverse local learning policies. Besides, building on advantage-weighted regression (AWR), we construct consistent local and global training objectives, significantly enhancing the efficiency and stability of FOVA. Further, we conduct an extensive theoretical analysis and rigorously show that the policy learned by FOVA enjoys strict policy improvement over the behavioral policy. Extensive experiments corroborate the significant performance gains of our proposed algorithm over existing baselines on widely used benchmarks.

Key facts. Cited by 5 papers in the map (data through 13 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 74 references that are in the map. First posted on arXiv 2 Dec 2025.

Builds on

44 of 74 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Federated Offline Policy Optimization with Dual Regularization24 May
  2. 2024Offline Reinforcement Learning: Role of State Aggregation and Trajectory Data25 Mar
  3. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices8 Feb
  4. 2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator7 Dec
  5. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL6 Oct
  6. 2023Federated Ensemble-Directed Offline Reinforcement Learning4 May
  7. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  8. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  9. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  10. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  11. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
3 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 29

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,653
  2. 2018Federated Optimization in Heterogeneous Networkscited by 2,981
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,272
  4. 2017Constrained Policy Optimizationcited by 931
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 767
  7. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 699
  8. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 182
  9. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 116
  10. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 80
  11. 2024NetLLM: Adapting Large Language Models for Networkingcited by 77
  12. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
17 more
  1. 2019Federated Transfer Reinforcement Learning for Autonomous Drivingcited by 44
  2. 2023The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyondcited by 35
  3. 2020When Deep Reinforcement Learning Meets Federated Learning: Intelligent Multi-Timescale Resource Management for Multi-access Edge Computing in 5G Ultra Dense Networkcited by 30
  4. 2024Finite-Time Analysis of On-Policy Heterogeneous Federated Reinforcement Learningcited by 30
  5. 2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
  6. 2021Kernel Continual Learningcited by 27
  7. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 26
  8. 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learningcited by 24
  9. 2023FedCache: A Knowledge Cache-driven Federated Learning Architecture for Personalized Edge Intelligencecited by 17
  10. 2024Distributed Deep Reinforcement Learning Based Gradient Quantization for Federated Learning Enabled Vehicle Edge Computingcited by 15
  11. 2022Model-Based Offline Meta-Reinforcement Learning with Regularizationcited by 14
  12. 2022Federated Offline Reinforcement Learningcited by 13
  13. 2018Deep Learning Based Caching for Self-Driving Car in Multi-access Edge Computingcited by 11
  14. 2022FedFormer: Contextual Federation with Attention in Reinforcement Learningcited by 8
  15. 2024Momentum for the Win: Collaborative Federated Reinforcement Learning across Heterogeneous Environmentscited by 8
  16. 2023PoPeC: PAoI-Centric Task Offloading with Priority over Unreliable Channelscited by 3
  17. 2023Client Selection for Federated Policy Optimization with Environment Heterogeneitycited by 3

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolutioncited by 12
  2. 2026FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectificationcited by 2
  3. 2026Less is More: Clustered Cross-Covariance Control for Offline RLcited by 1
  4. 2026AdamO: A Collapse-Suppressed Optimizer for Offline RLcited by 1
  5. 2026Cloud-Edge Collaborative Large Models for Robust Photovoltaic Power Forecastingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Federated Offline Policy Learning
  2. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  3. 2022Leveraging Offline Data in Online Reinforcement Learning
  4. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning
  5. 2026FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learning
  6. 2026Active Offline-to-Online Reinforcement Learning
  7. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  8. 2023Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees