Data to 7 Oct 2026

Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning

arXiv 2202.11566 ↗First posted 23 Feb 2022cs.LGcited by 125

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Reinforcement Learning (RL) aims to learn policies from previously collected datasets without exploring the environment. Directly applying off-policy algorithms to offline RL usually fails due to the extrapolation error caused by the out-of-distribution (OOD) actions. Previous methods tackle such problem by penalizing the Q-values of OOD actions or constraining the trained policy to be close to the behavior policy. Nevertheless, such methods typically prevent the generalization of value functions beyond the offline data and also lack precise characterization of OOD data. In this paper, we propose Pessimistic Bootstrapping for offline RL (PBRL), a purely uncertainty-driven offline algorithm without explicit policy constraints. Specifically, PBRL conducts uncertainty quantification via the disagreement of bootstrapped Q-functions, and performs pessimistic updates by penalizing the value function based on the estimated uncertainty. To tackle the extrapolating error, we further propose a novel OOD sampling method. We show that such OOD sampling and pessimistic bootstrapping yields provable uncertainty quantifier in linear MDPs, thus providing the theoretical underpinning for PBRL. Extensive experiments on D4RL benchmark show that PBRL has better performance compared to the state-of-the-art algorithms.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 56 of its 65 references that are in the map. First posted on arXiv 23 Feb 2022.

Builds on

56 of 65 references.

Closest ancestors · 25

Same direction, newest first

13 more

Other references · 31

2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647

19 more

2016Deep Exploration via Bootstrapped DQNcited by 518
2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
2017Minimax Regret Bounds for Reinforcement Learningcited by 500
2020Planning to Explore via Self-Supervised World Modelscited by 365
2019Reinforcement Learning in Healthcare: A Surveycited by 308
2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
2018Learning to Navigate in Cities Without a Mapcited by 103
2020Information Theoretic Regret Bounds for Online Nonlinear Controlcited by 87
2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
2017The Uncertainty Bellman Equation and Explorationcited by 81
2020Active Learning for Nonlinear System Identification with Guaranteescited by 78
2018Efficient Exploration through Bayesian Deep Q-Networkscited by 67
2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
2019Distributional Reinforcement Learning for Efficient Explorationcited by 59
2018Information-Directed Exploration for Deep Reinforcement Learningcited by 52
2021Principled Exploration via Optimistic Bootstrapping and Backward Inductioncited by 26
2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 38; 2024: 33; 2025: 28; 2026: 11.

125 papers.

Sort:

By area:

2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 55
2023Behavior Proximal Policy Optimizationcited by 43
2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
2022A Review of Uncertainty for Deep Reinforcement Learningcited by 36
2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31

38 more

2025Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failurescited by 30
2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learningcited by 29
2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learningcited by 25
2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Beliefcited by 22
2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 20
2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learningcited by 18
2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppressioncited by 17
2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learningcited by 15
2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operatorscited by 14
2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
2023Uncertainty-driven Trajectory Truncation for Data Augmentation in Offline Reinforcement Learningcited by 13
2024Off-Policy Primal-Dual Safe Reinforcement Learningcited by 13
2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learningcited by 12
2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothnesscited by 12
2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learningcited by 12
2024Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learningcited by 10
2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guaranteescited by 10
2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 9
2023Offline Minimax Soft-Q-learning Under Realizability and Partial Coveragecited by 9
2023Conservative State Value Estimation for Offline Reinforcement Learningcited by 9
2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operatorcited by 9
2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
2024Exclusively Penalized Q-learning for Offline Reinforcement Learningcited by 8
2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way