mlsanityData to 7 Oct 2026

The Importance of Pessimism in Fixed-Dataset Policy Optimization

arXiv 2009.06799 ↗First posted 15 Sep 2020cs.AI cs.LGcited by 99

Not on the map. Why?

Cited by 71 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study worst-case guarantees on the expected return of fixed-dataset policy optimization algorithms. Our core contribution is a unified conceptual and mathematical framework for the study of algorithms in this regime. This analysis reveals that for naive approaches, the possibility of erroneous value overestimation leads to a difficult-to-satisfy requirement: in order to guarantee that we select a policy which is near-optimal, we may need the dataset to be informative of the value of every policy. To avoid this, algorithms can follow the pessimism principle, which states that we should choose the policy which acts optimally in the worst possible world. We show why pessimistic algorithms can achieve good performance even when the dataset is not informative of every policy, and derive families of algorithms which follow this principle. These theoretical findings are validated by experiments on a tabular gridworld, and deep learning experiments on four MinAtar environments.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 25; 2022: 28; 2023: 26; 2024: 7; 2025: 7; 2026: 4.

99 papers.

Sort:

By area:

  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  2. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  3. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  4. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  5. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  6. 2021Offline RL Without Off-Policy Evaluationcited by 150
  7. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
  8. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  9. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  10. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  11. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  12. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 77
38 more
  1. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  2. 2021Risk-Averse Offline Reinforcement Learningcited by 65
  3. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 59
  4. 2021Corruption-Robust Offline Reinforcement Learningcited by 51
  5. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  6. 2021Offline Reinforcement Learning as Anti-Explorationcited by 44
  7. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  8. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  9. 2021Model-Based Offline Planning with Trajectory Pruningcited by 30
  10. 2021Offline Reinforcement Learning with Pseudometric Learningcited by 28
  11. 2021On the Optimality of Batch Policy Optimization Algorithmscited by 28
  12. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  13. 2021Continuous Doubly Constrained Batch Reinforcement Learningcited by 27
  14. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
  15. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processescited by 26
  16. 2021Offline Neural Contextual Bandits: Pessimism, Optimization and Generalizationcited by 25
  17. 2021Can Reinforcement Learning Find Stackelberg-Nash Equilibria in General-Sum Markov Games with Myopic Followers?cited by 24
  18. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasetscited by 24
  19. 2022Boosting Offline Reinforcement Learning via Data Rebalancingcited by 22
  20. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weightingcited by 22
  21. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RLcited by 21
  22. 2023Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systemscited by 19
  23. 2020DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPscited by 16
  24. 2022Robust Anytime Learning of Markov Decision Processescited by 16
  25. 2022Agent-Controller Representations: Principled Offline RL with Rich Exogenous Informationcited by 15
  26. 2023Off-Policy Evaluation for Large Action Spaces via Policy Convolutioncited by 14
  27. 2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
  28. 2023Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidancecited by 13
  29. 2024Offline Multi-task Transfer RL with Representational Penalizationcited by 13
  30. 2022Behavior Prior Representation learning for Offline Reinforcement Learningcited by 12
  31. 2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learningcited by 12
  32. 2023Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problemscited by 12
  33. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coveragecited by 12
  34. 2021Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learningcited by 10
  35. 2022Robust Imitation Learning from Corrupted Demonstrationscited by 10
  36. 2022Bellman Residual Orthogonalization for Offline Reinforcement Learningcited by 10
  37. 2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarkscited by 10
  38. 2022Multi-Task Off-Policy Learning from Bandit Feedbackcited by 9

50 most cited of 99 All 99 (JSON)