mlsanityData to 14 Oct 2026

Provably Efficient Offline Reinforcement Learning with Perturbed Data Sources

arXiv 2306.08364 ↗First posted 14 Jun 2023stat.ML cs.IT cs.LGcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Existing theoretical studies on offline reinforcement learning (RL) mostly consider a dataset sampled directly from the target task. In practice, however, data often come from several heterogeneous but related sources. Motivated by this gap, this work aims at rigorously understanding offline RL with multiple datasets that are collected from randomly perturbed versions of the target task instead of from itself. An information-theoretic lower bound is derived, which reveals a necessary requirement on the number of involved sources in addition to that on the number of data samples. Then, a novel HetPEVI algorithm is proposed, which simultaneously considers the sample uncertainties from a finite number of data samples per data source and the source uncertainties due to a finite number of available data sources. Theoretical analyses demonstrate that HetPEVI can solve the target task as long as the data sources collectively provide a good data coverage. Moreover, HetPEVI is demonstrated to be optimal up to a polynomial factor of the horizon length. Finally, the study is extended to offline Markov games and offline robust RL, which demonstrates the generality of the proposed designs and theoretical analyses.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 57 references that are in the map. First posted on arXiv 14 Jun 2023.

Builds on

43 of 57 references.

Closest ancestors · 17

Same direction, newest first

  1. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximation14 Sep
  2. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity11 Aug
  3. 2022Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonus1 Jun
  4. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game31 May
  5. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning11 Apr
  6. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism11 Mar
  7. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity28 Feb
  8. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  9. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  10. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  11. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  12. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
5 more
  1. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  2. 2020DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPs18 Oct
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020MOPO: Model-based Offline Policy Optimization27 May
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 26

  1. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  2. 2018Is Q-learning Provably Efficient?cited by 487
  3. 2021Reinforcement learning based recommender systems: A surveycited by 188
  4. 2022Robust Reinforcement Learning using Offline Datacited by 96
  5. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 93
  6. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  7. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 83
  8. 2021RL for Latent MDPs: Regret Guarantees and a Lower Boundcited by 70
  9. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  10. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
  11. 2021Federated Multi-Armed Banditscited by 59
  12. 2020Impact of Representation Learning in Linear Banditscited by 52
14 more
  1. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 44
  2. 2022Provable Benefits of Representational Transfer in Reinforcement Learningcited by 38
  3. 2019Federated Deep Reinforcement Learningcited by 35
  4. 2021On the Power of Multitask Representation Learning in Linear MDPcited by 30
  5. 2022When is Offline Two-Player Zero-Sum Markov Game Solvable?cited by 28
  6. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  7. 2021Provably Efficient Cooperative Multi-Agent Reinforcement Learning with Function Approximationcited by 23
  8. 2021Provably Efficient Multi-Task Reinforcement Learning with Model Transfercited by 16
  9. 2022Nearly Minimax Algorithms for Linear Bandits with Shared Representationcited by 16
  10. 2022Model-Based Offline Meta-Reinforcement Learning with Regularizationcited by 15
  11. 2022Federated Offline Reinforcement Learningcited by 13
  12. 2022Model-Based Reinforcement Learning for Offline Zero-Sum Markov Gamescited by 11
  13. 2022Calibrated inference: statistical inference that accounts for both sampling uncertainty and distributional uncertaintycited by 5
  14. 2021Random Effect Banditscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 0; 2026: 1.

3 papers.

  1. 2024Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Sufficescited by 11
  2. 2023Advancing RAN Slicing with Offline Reinforcement Learningcited by 5
  3. 2026Offline Learning of Nash Stable Coalition Structures with Possibly Overlapping Coalitionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  2. 2022Leveraging Offline Data in Online Reinforcement Learning
  3. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  4. 2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning
  5. 2025Augmenting Online RL with Offline Data is All You Need: A Unified Hybrid RL Algorithm Design and Analysis
  6. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability
  7. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism
  8. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory