Data to 14 Oct 2026

Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters

arXiv 2205.13703 ↗First posted 27 May 2022cs.LGcited by 76

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Motivated by the success of ensembles for uncertainty estimation in supervised learning, we take a renewed look at how ensembles of Q-functions can be leveraged as the primary source of pessimism for offline reinforcement learning (RL). We begin by identifying a critical flaw in a popular algorithmic choice used by many ensemble-based RL algorithms, namely the use of shared pessimistic target values when computing each ensemble member's Bellman error. Through theoretical analyses and construction of examples in toy MDPs, we demonstrate that shared pessimistic targets can paradoxically lead to value estimates that are effectively optimistic. Given this result, we propose MSG, a practical offline RL algorithm that trains an ensemble of Q-functions with independently computed targets based on completely separate networks, and optimizes a policy with respect to the lower confidence bound of predicted action values. Our experiments on the popular D4RL and RL Unplugged offline RL benchmarks demonstrate that on challenging domains such as antmazes, MSG with deep ensembles surpasses highly well-tuned state-of-the-art methods by a wide margin. Additionally, through ablations on benchmarks domains, we verify the critical significance of using independently trained Q-functions, and study the role of ensemble size. Finally, as using separate networks per ensemble member can become computationally costly with larger neural network architectures, we investigate whether efficient ensemble approximations developed for supervised learning can be similarly effective, and demonstrate that they do not match the performance and robustness of MSG with separate networks, highlighting the need for new efforts into efficient uncertainty estimation directed at RL.

Key facts. Cited by 76 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 54 of its 78 references that are in the map. First posted on arXiv 27 May 2022.

Builds on

54 of 78 references.

Closest ancestors · 15

Same direction, newest first

3 more

Other references · 39

2017Attention Is All You Needcited by 60,361
2014Adam: A Method for Stochastic Optimizationcited by 37,240
2017On Calibration of Modern Neural Networkscited by 4,960
2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
2016Identity Mappings in Deep Residual Networkscited by 1,965
2016OpenAI Gymcited by 1,736
2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
2018Soft Actor-Critic Algorithms and Applicationscited by 1,240

27 more

2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
2015Gated Graph Sequence Neural Networkscited by 796
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
2017Deep Neural Networks as Gaussian Processescited by 622
2016Deep Exploration via Bootstrapped DQNcited by 527
2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
2017Count-Based Exploration with Neural Density Modelscited by 257
2020Feature Learning in Infinite-Width Neural Networkscited by 190
2020Training independent subnetworks for robust predictioncited by 177
2018Deep Reinforcement Learning and the Deadly Triadcited by 174
2017Deep Exploration via Randomized Value Functionscited by 156
2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
2019Neural Tangents: Fast and Easy Infinite Neural Networks in Pythoncited by 149
2015Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networkscited by 139
2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variablescited by 50
2018Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?cited by 39
2020Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weightingcited by 38
2020Accountable Off-Policy Evaluation With Kernel Bellman Statisticscited by 36
2020Hydra: Preserving Ensemble Diversity for Model Distillationcited by 31
2020Offline Policy Selection under Uncertaintycited by 31
2017Implicit Weight Uncertainty in Neural Networkscited by 30
2020Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluationcited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 18; 2024: 24; 2025: 14; 2026: 13.

76 papers.

Sort:

By area:

2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 119
2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 84
2022Offline RL Policies Should be Trained to be Adaptivecited by 52
2023Anti-Exploration by Random Network Distillationcited by 50
2022A Review of Uncertainty for Deep Reinforcement Learningcited by 40
2026Q-learning with Adjoint Matchingcited by 37
2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 33

38 more

2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 31
2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learningcited by 31
2022Controlling Commercial Cooling Systems Using Reinforcement Learningcited by 29
2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 21
2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
2024Off-Policy Primal-Dual Safe Reinforcement Learningcited by 14
2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learningcited by 12
2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothnesscited by 12
2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 11
2024Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learningcited by 10
2023Off-the-Grid MARL: Datasets with Baselines for Offline Multi-Agent Reinforcement Learningcited by 8
2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptionscited by 8
2022Epistemic Monte Carlo Tree Searchcited by 7
2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learningcited by 7
2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
2026ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Trainingcited by 7
2023Decoupled Prioritized Resampling for Offline RLcited by 6
2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 6
2025Uncertainty in Action: Confidence Elicitation in Embodied Agentscited by 5
2023Budgeting Counterfactual for Offline RLcited by 4
2024Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?cited by 4
2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learningcited by 4
2026Drift Q-Learningcited by 4
2023VIPeR: Provably Efficient Algorithm for Offline RL with Neural Function Approximationcited by 3
2024Distilling LLMs' Decomposition Abilities into Compact Language Modelscited by 3
2024Offline Reinforcement Learning with Behavioral Supervisor Tuningcited by 3
2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spacescited by 3
2025Provable Zero-Shot Generalization in Offline Reinforcement Learningcited by 3
2025Real-Time Diffusion Policies for Games: Enhancing Consistency Policies with Q-Ensemblescited by 3
2025Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhoodcited by 3
2026Uncertainty-Aware Rank-One MIMO Q Network Framework for Accelerated Offline Reinforcement Learningcited by 3

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way