mlsanityData to 7 Oct 2026

Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage

arXiv 2107.06226 ↗First posted 13 Jul 2021cs.LG cs.AI stat.MLcited by 160

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We study model-based offline Reinforcement Learning with general function approximation without a full coverage assumption on the offline data distribution. We present an algorithm named Constrained Pessimistic Policy Optimization (CPPO)which leverages a general function class and uses a constraint over the model class to encode pessimism. Under the assumption that the ground truth model belongs to our function class (i.e., realizability in the function class), CPPO has a PAC guarantee with offline data only providing partial coverage, i.e., it can learn a policy that competes against any policy that is covered by the offline data. We then demonstrate that this algorithmic framework can be applied to many specialized Markov Decision Processes where additional structural assumptions can further refine the concept of partial coverage. Two notable examples are: (1) low-rank MDP with representation learning where the partial coverage condition is defined using a relative condition number measured by the unknown ground truth feature representation; (2) factored MDP where the partial coverage condition is defined using density ratio based concentrability coefficients associated with individual factors.

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 50 of its 81 references that are in the map. First posted on arXiv 13 Jul 2021.

Builds on

50 of 81 references.

Closest ancestors · 19

Same direction, newest first

  1. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL16 Jun
  2. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  3. 2021Corruption-Robust Offline Reinforcement Learning11 Jun
  4. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  5. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learning25 Mar
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  7. 2021Continuous Doubly Constrained Batch Reinforcement Learning18 Feb
  8. 2021Finite Sample Analysis of Minimax Offline Reinforcement Learning: Completeness, Fast Rates and First-Order Efficiency5 Feb
  9. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction2 Feb
  10. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  11. 2020Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient8 Nov
  12. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
7 more
  1. 2020Batch Value-function Approximation with Only Realizability11 Aug
  2. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020MOPO: Model-based Offline Policy Optimization27 May
  5. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 31

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  3. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  4. 2013Learning to Optimize Via Posterior Samplingcited by 253
  5. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  6. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  7. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  8. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  9. 2020Nearly Minimax Optimal Reinforcement Learning for Linear Mixture Markov Decision Processescited by 199
  10. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  11. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 181
  12. 2013Finite-Time Analysis of Kernelised Contextual Banditscited by 177
19 more
  1. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  2. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  3. 2015On the Equivalence between Kernel Quadrature Rules and Random Feature Expansionscited by 136
  4. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  5. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  6. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  7. 2019Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensemblescited by 100
  8. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  9. 2020Toward the Fundamental Limits of Imitation Learningcited by 96
  10. 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 91
  11. 2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 87
  12. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 83
  13. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  14. 2019Corruption-robust exploration in episodic reinforcement learningcited by 72
  15. 2020A Unifying View of Optimism in Episodic Reinforcement Learningcited by 50
  16. 2021Leveraging Good Representations in Linear Contextual Banditscited by 29
  17. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
  18. 2020Stable Policy Optimization via Off-Policy Divergence Regularizationcited by 16
  19. 2013Policy Iteration for Factored MDPscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 31; 2023: 51; 2024: 43; 2025: 18; 2026: 17.

160 papers.

Sort:

By area:

  1. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  2. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  3. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  4. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  5. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92
  6. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  7. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  8. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  9. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
  10. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  11. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Gamecited by 53
  12. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
38 more
  1. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  2. 2023Making RL with Preference-based Feedback Efficient via Randomizationcited by 46
  3. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 42
  4. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  5. 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 41
  6. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  7. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  8. 2023Adversarial Model for Offline Reinforcement Learningcited by 31
  9. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 30
  10. 2022Provably Efficient Offline Multi-agent Reinforcement Learning via Strategy-wise Bonuscited by 29
  11. 2022Guarantees for Epsilon-Greedy Reinforcement Learning with Function Approximationcited by 29
  12. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
  13. 2022When is Offline Two-Player Zero-Sum Markov Game Solvable?cited by 27
  14. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
  15. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  16. 2022Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learningcited by 26
  17. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  18. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  19. 2024Sample-Efficient Robust Multi-Agent Reinforcement Learning in the Face of Environmental Uncertaintycited by 24
  20. 2024Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Modelscited by 24
  21. 2024More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learningcited by 23
  22. 2022On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximationcited by 22
  23. 2022Future-Dependent Value-Based Off-Policy Evaluation in POMDPscited by 20
  24. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  25. 2023Distributional Offline Policy Evaluation with Predictive Error Guaranteescited by 19
  26. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
  27. 2022Pessimism for Offline Linear Contextual Bandits using ℓp Confidence Setscited by 18
  28. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  29. 2022Learning Bellman Complete Representations for Offline Policy Evaluationcited by 17
  30. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPscited by 17
  31. 2022On the Role of Discount Factor in Offline Reinforcement Learningcited by 16
  32. 2023Offline Learning in Markov Games with General Function Approximationcited by 16
  33. 2023Model-based Offline Reinforcement Learning with Count-based Conservatismcited by 16
  34. 2025Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspectivecited by 16
  35. 2023Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Banditscited by 14
  36. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  37. 2024Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inferencecited by 14
  38. 2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14

50 most cited of 160 All 160 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes
  2. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism
  3. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  4. 2024Optimistic Model Rollouts for Pessimistic Offline Policy Optimization
  5. 2023Model-based Offline Reinforcement Learning with Local Misspecification
  6. 2022State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learning
  7. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learning
  8. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism