mlsanityData to 7 Oct 2026

Offline Reinforcement Learning with Additional Covering Distributions

arXiv 2305.12679 ↗First posted 22 May 2023cs.LG stat.MLcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We study learning optimal policies from a logged dataset, i.e., offline RL, with function approximation. Despite the efforts devoted, existing algorithms with theoretic finite-sample guarantees typically assume exploratory data coverage or strong realizable function classes, which is hard to be satisfied in reality. While there are recent works that successfully tackle these strong assumptions, they either require the gap assumptions that only could be satisfied by part of MDPs or use the behavior regularization that makes the optimality of learned policy even intractable. To solve this challenge, we provide finite-sample guarantees for a simple algorithm based on marginalized importance sampling (MIS), showing that sample-efficient offline RL for general MDPs is possible with only a partial coverage dataset and weak realizable function classes given additional side information of a covering distribution. Furthermore, we demonstrate that the covering distribution trades off prior knowledge of the optimal trajectories against the coverage requirement of the dataset, revealing the effect of this inductive bias in the learning processes.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 44 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

28 of 44 references.

Closest ancestors · 18

Same direction, newest first

  1. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare2 May
  2. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning30 Jan
  3. 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraints28 Dec
  4. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian1 Nov
  5. 2022On Gap-dependent Bounds for Offline Reinforcement Learning1 Jun
  6. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gaps25 Mar
  7. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  8. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  9. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation21 Nov
  10. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  11. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  12. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
6 more
  1. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  2. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning10 Mar
  3. 2021Instabilities of Offline RL with Pre-Trained Neural Representation8 Mar
  4. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  5. 2020Batch Value-function Approximation with Only Realizability11 Aug
  6. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul

Other references · 10

  1. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  2. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  3. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  4. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  5. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  6. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  7. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  8. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  9. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  10. 2022Beyond the Return: Off-policy Function Estimation under User-specified Error-measuring Distributionscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  2. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?
  3. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning
  4. 2023What can online reinforcement learning with function approximation benefit from general coverage conditions?
  5. 2022The Role of Coverage in Online Reinforcement Learning
  6. 2026On the Complexity of Offline Reinforcement Learning with Q^star-Approximation and Partial Coverage
  7. 2025Optimal Single-Policy Sample Complexity and Transient Coverage for Average-Reward Offline RL
  8. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples