mlsanityData to 7 Oct 2026

Federated Offline Policy Learning

arXiv 2305.12407 ↗First posted 21 May 2023cs.LG cs.DC econ.EMcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We consider the problem of learning personalized decision policies from observational bandit feedback data across multiple heterogeneous data sources. In our approach, we introduce a novel regret analysis that establishes finite-sample upper bounds on distinguishing notions of global regret for all data sources on aggregate and of local regret for any given data source. We characterize these regret bounds by expressions of source heterogeneity and distribution shift. Moreover, we examine the practical considerations of this problem in the federated setting where a central server aims to train a policy on data distributed across the heterogeneous sources without collecting any of their raw data. We present a policy learning algorithm amenable to federation based on the aggregation of local policies trained with doubly robust offline policy evaluation strategies. Our analysis and supporting experimental results provide insights into tradeoffs in the participation of heterogeneous data sources in offline policy learning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 44 references that are in the map. First posted on arXiv 21 May 2023.

Builds on

27 of 44 references.

References · 27

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  5. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  6. 2019Agnostic Federated Learningcited by 451
  7. 2014Taming the Monster: A Fast and Simple Algorithm for Contextual Banditscited by 200
  8. 2019Federated Evaluation of On-device Personalizationcited by 147
  9. 2020Bypassing the Monster: A Faster and Simpler Optimal Algorithm for Contextual Bandits under Realizabilitycited by 114
  10. 2020Differentially-Private Federated Linear Banditscited by 83
  11. 2018A Contextual Bandit Bake-offcited by 64
  12. 2018Offline Multi-Action Policy Learning: Generalization and Optimizationcited by 64
15 more
  1. 2021Federated Linear Contextual Banditscited by 61
  2. 2017Balanced Policy Evaluation and Learningcited by 51
  3. 2021Optimality and Stability in Federated Learning: A Game-theoretic Approachcited by 40
  4. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  5. 2020Optimal Off-Policy Evaluation from Multiple Logging Policiescited by 30
  6. 2021Federated Causal Inference in Heterogeneous Observational Datacited by 24
  7. 2020Federated Residual Learningcited by 23
  8. 2021Adapting to Misspecification in Contextual Bandits with Offline Regression Oraclescited by 21
  9. 2021Policy Learning with Adaptively Collected Datacited by 21
  10. 2017Effective Evaluation using Logged Bandit Feedback from Multiple Loggerscited by 20
  11. 2020A Theory of Multiple-Source Adaptation with Limited Target Labeled Datacited by 18
  12. 2022Federated Offline Reinforcement Learningcited by 13
  13. 2022Multi-Task Off-Policy Learning from Bandit Feedbackcited by 9
  14. 2019Off-policy Learning for Multiple Loggerscited by 4
  15. 2023An Empirical Evaluation of Federated Contextual Bandit Algorithmscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Robust Offline Policy Learning with Observational Data from Multiple Sources
  2. 2025FOVA: Offline Federated Reinforcement Learning with Mixed-Quality Data
  3. 2025Reinforcement Learning for Individual Optimal Policy from Heterogeneous Data
  4. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices
  5. 2024Federated Offline Policy Optimization with Dual Regularization
  6. 2024Is Offline Decision Making Possible with Only Few Samples? Reliable Decisions in Data-Starved Bandits via Trust Region Enhancement
  7. 2021Offline Neural Contextual Bandits: Pessimism, Optimization and Generalization
  8. 2026Offline-to-Online Learning in Linear Bandits