mlsanityData to 7 Oct 2026

Federated Offline Policy Optimization with Dual Regularization

arXiv 2405.17474 ↗First posted 24 May 2024cs.LG cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Federated Reinforcement Learning (FRL) has been deemed as a promising solution for intelligent decision-making in the era of Artificial Internet of Things. However, existing FRL approaches often entail repeated interactions with the environment during local updating, which can be prohibitively expensive or even infeasible in many real-world domains. To overcome this challenge, this paper proposes a novel offline federated policy optimization algorithm, named textttDRPO, which enables distributed agents to collaboratively learn a decision policy only from private and static data without further environmental interactions. textttDRPO leverages dual regularization, incorporating both the local behavioral policy and the global aggregated policy, to judiciously cope with the intrinsic two-tier distributional shifts in offline FRL. Theoretical analysis characterizes the impact of the dual regularization on performance, demonstrating that by achieving the right balance thereof, textttDRPO can effectively counteract distributional shifts and ensure strict policy improvement in each federative learning round. Extensive experiments validate the significant performance gains of textttDRPO over baseline methods.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 32 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

21 of 32 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Federated Ensemble-Directed Offline Reinforcement Learning4 May
  2. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020MOPO: Model-based Offline Policy Optimization27 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 14

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2019Advances and Open Problems in Federated Learningcited by 3,021
  5. 2019Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systemscited by 77
  6. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 52
  7. 2019Federated Transfer Reinforcement Learning for Autonomous Drivingcited by 43
  8. 2019Federated Deep Reinforcement Learningcited by 34
  9. 2020When Deep Reinforcement Learning Meets Federated Learning: Intelligent Multi-Timescale Resource Management for Multi-access Edge Computing in 5G Ultra Dense Networkcited by 29
  10. 2022Federated Stochastic Approximation under Markov Noise and Heterogeneity: Applications in Reinforcement Learningcited by 27
  11. 2022Model-Based Offline Meta-Reinforcement Learning with Regularizationcited by 14
  12. 2021Multi-Task Federated Reinforcement Learning with Adversariescited by 13
2 more
  1. 2022Federated Offline Reinforcement Learningcited by 13
  2. 2019Federated Reinforcement Distillation with Proxy Experience Memorycited by 5

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Datacited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2026FORLER: Federated Offline Reinforcement Learning with Q-Ensemble and Actor Rectification
  2. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices
  3. 2023Federated Offline Policy Learning
  4. 2026Distributed Direct Preference Optimization
  5. 2026FedGuide: Diffusion Prior Alignment and Value Baseline Guidance for Heterogeneous Federated Reinforcement Learning
  6. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization
  7. 2026FGRPO: Federated GRPO with Adaptive Aggregation on Non-IID Data
  8. 2022Offline Policy Optimization in RL with Variance Regularizaton