mlsanityData to 14 Oct 2026

Federated Offline Reinforcement Learning

arXiv 2206.05581 ↗First posted 11 Jun 2022stat.ML cs.LG stat.MEcited by 13

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Evidence-based or data-driven dynamic treatment regimes are essential for personalized medicine, which can benefit from offline reinforcement learning (RL). Although massive healthcare data are available across medical institutions, they are prohibited from sharing due to privacy constraints. Besides, heterogeneity exists in different sites. As a result, federated offline RL algorithms are necessary and promising to deal with the problems. In this paper, we propose a multi-site Markov decision process model that allows for both homogeneous and heterogeneous effects across sites. The proposed model makes the analysis of the site-level features possible. We design the first federated policy optimization algorithm for offline RL with sample complexity. The proposed algorithm is communication-efficient, which requires only a single round of communication interaction by exchanging summary statistics. We give a theoretical guarantee for the proposed algorithm, where the suboptimality for the learned policies is comparable to the rate as if data is not distributed. Extensive simulations demonstrate the effectiveness of the proposed algorithm. The method is applied to a sepsis dataset in multiple sites to illustrate its use in clinical settings.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 83 references that are in the map. First posted on arXiv 11 Jun 2022.

Builds on

36 of 83 references.

References · 36

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  6. 2018Federated Learning for Mobile Keyboard Predictioncited by 572
  7. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  8. 2019Federated Machine Learning: Concept and Applicationscited by 412
  9. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  10. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  11. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  12. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
24 more
  1. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  2. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 179
  3. 2019Privacy-preserving Federated Brain Tumour Segmentationcited by 169
  4. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
  5. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  6. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  7. 2020Batch Value-function Approximation with Only Realizabilitycited by 104
  8. 2017Continuous State-Space Models for Optimal Sepsis Treatment - a Deep Reinforcement Learning Approachcited by 68
  9. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  10. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  11. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
  12. 2021Fault-Tolerant Federated Reinforcement Learning with Theoretical Guaranteecited by 54
  13. 2017A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Unitscited by 50
  14. 2019Federated Deep Reinforcement Learningcited by 35
  15. 2019GADMM: Fast and Communication Efficient Framework for Distributed Machine Learningcited by 30
  16. 2022FedKL: Tackling Data Heterogeneity in Federated Reinforcement Learning by Penalizing KL Divergencecited by 29
  17. 2019Heterogeneity-aware and communication-efficient distributed statistical inferencecited by 15
  18. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluationcited by 14
  19. 2018Distributed Inference for Linear Support Vector Machinecited by 13
  20. 2022Byzantine-Robust Online and Offline Distributed Reinforcement Learningcited by 13
  21. 2018Communication-Efficient Policy Gradient Methods for Distributed Reinforcement Learningcited by 11
  22. 2018DeepMood: Modeling Mobile Phone Typing Dynamics for Mood Detectioncited by 10
  23. 2022Byzantine-Robust Federated Linear Banditscited by 8
  24. 2020Semi-Supervised Off Policy Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 3; 2025: 3; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Sufficescited by 11
  2. 2023Federated Ensemble-Directed Offline Reinforcement Learningcited by 6
  3. 2025FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Datacited by 5
  4. 2023Client Selection for Federated Policy Optimization with Environment Heterogeneitycited by 3
  5. 2023Provably Efficient Offline Reinforcement Learning with Perturbed Data Sourcescited by 3
  6. 2024FairFML: Fair Federated Machine Learning with a Case Study on Reducing Gender Disparities in Cardiac Arrest Outcome Predictioncited by 3
  7. 2025Beyond Prediction: Reinforcement Learning as the Defining Leap in Healthcare AIcited by 3
  8. 2024Federated Offline Policy Optimization with Dual Regularizationcited by 1
  9. 2026Adaptive Estimation and Optimal Control in Offline Contextual MDPs without Stationaritycited by 1
  10. 2023Federated Offline Policy Learningcited by 0
  11. 2025Latent Factor Point Processes for Patient Representation in Electronic Health Recordscited by 0
  12. 2026Learning Sequential Decisions from Multiple Sources via Group-Robust Markov Decision Processescited by 0
1 more
  1. 2026Privacy Preserving Reinforcement Learning with One-Sided Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Semi-Markov Offline Reinforcement Learning for Healthcare
  2. 2024Reinforcement Learning in Dynamic Treatment Regimes Needs Critical Reexamination
  3. 2025Reinforcement Learning for Individual Optimal Policy from Heterogeneous Data
  4. 2023Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications
  5. 2022Offline Policy Evaluation and Optimization under Confounding
  6. 2022Learning Optimal Treatment Strategies for Sepsis Using Offline Reinforcement Learning in Continuous Space
  7. 2025Hybrid Adaptive Conformal Offline Reinforcement Learning for Fair Population Health Management
  8. 2025Test-Time Learning and Inference-Time Deliberation for Efficiency-First Offline Reinforcement Learning in Care Coordination and Population Health Management