mlsanityData to 14 Oct 2026

Bellman-consistent Pessimism for Offline Reinforcement Learning

arXiv 2106.06926 ↗First posted 13 Jun 2021cs.LG cs.AI stat.MLcited by 269

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The use of pessimism, when reasoning about datasets lacking exhaustive exploration has recently gained prominence in offline reinforcement learning. Despite the robustness it adds to the algorithm, overly pessimistic reasoning can be equally damaging in precluding the discovery of good policies, which is an issue for the popular bonus-based pessimism. In this paper, we introduce the notion of Bellman-consistent pessimism for general function approximation: instead of calculating a point-wise lower bound for the value function, we implement pessimism at the initial state over the set of functions consistent with the Bellman equations. Our theoretical guarantees only require Bellman closedness as standard in the exploratory setting, in which case bonus-based pessimism fails to provide guarantees. Even in the special case of linear function approximation where stronger expressivity assumptions hold, our result improves upon a recent bonus-based approach by O(d) in its sample complexity when the action space is finite. Remarkably, our algorithms automatically adapt to the best bias-variance tradeoff in the hindsight, whereas most prior approaches require tuning extra hyperparameters a priori.

Key facts. Cited by 269 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 49 references that are in the map. First posted on arXiv 13 Jun 2021.

Builds on

22 of 49 references.

Closest ancestors · 9

Same direction, newest first

  1. 2021Instabilities of Offline RL with Pre-Trained Neural Representation8 Mar
  2. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  3. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  4. 2020Batch Value-function Approximation with Only Realizability11 Aug
  5. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020MOPO: Model-based Offline Policy Optimization27 May
  8. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 13

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  3. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  4. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 292
  5. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  6. 2019A Theory of Regularized Markov Decision Processescited by 286
  7. 2019Provably Efficient Exploration in Policy Optimizationcited by 205
  8. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  9. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  10. 2020Optimistic Policy Optimization with Bandit Feedbackcited by 79
  11. 2020Exponential Lower Bounds for Planning in MDPs With Linearly-Realizable Optimal Action-Value Functionscited by 71
  12. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
1 more
  1. 2014Approximate Policy Iteration Schemes: A Comparisoncited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 62; 2023: 67; 2024: 66; 2025: 38; 2026: 25.

269 papers.

Sort:

By area:

  1. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  2. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  3. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  4. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  5. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  6. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  7. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  8. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  9. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  10. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 129
  11. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  12. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
38 more
  1. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  2. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  3. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  4. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 110
  5. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  6. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 98
  7. 2022Robust Reinforcement Learning using Offline Datacited by 96
  8. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 95
  9. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 89
  10. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 85
  11. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 84
  12. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
  13. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  14. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 76
  15. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 76
  16. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  17. 2024Robust Preference Optimization through Reward Model Distillationcited by 67
  18. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 65
  19. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  20. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Gamecited by 55
  21. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  22. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  23. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  24. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
  25. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  26. 2024Transfer Q Star: Principled Decoding for LLM Alignmentcited by 48
  27. 2024Towards Efficient LLM Grounding for Embodied Multi-Agent Collaborationcited by 45
  28. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 44
  29. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  30. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 37
  31. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learningcited by 36
  32. 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34
  33. 2022On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluationcited by 33
  34. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 33
  35. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  36. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 32
  37. 2022Policy learning "without" overlap: Pessimism and generalized empirical Bernstein's inequalitycited by 32
  38. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31

50 most cited of 269 All 269 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Generalization in offline RL: The structure is more important than the amount of pessimism
  2. 2025Bellman Calibration for V-Learning in Offline Reinforcement Learning
  3. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learning
  4. 2025Semi-pessimistic Reinforcement Learning
  5. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  6. 2020Pessimism About Unknown Unknowns Inspires Conservatism
  7. 2024Linear Bellman Completeness Suffices for Efficient Online Reinforcement Learning with Few Actions
  8. 2024The Virtues of Pessimism in Inverse Reinforcement Learning