mlsanityData to 7 Oct 2026

Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble

arXiv 2110.01548 ↗First posted 4 Oct 2021cs.LG cs.AIcited by 297

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (offline RL), which aims to find an optimal policy from a previously collected static dataset, bears algorithmic difficulties due to function approximation errors from out-of-distribution (OOD) data points. To this end, offline RL algorithms adopt either a constraint or a penalty term that explicitly guides the policy to stay close to the given dataset. However, prior methods typically require accurate estimation of the behavior policy or sampling from OOD data points, which themselves can be a non-trivial problem. Moreover, these methods under-utilize the generalization ability of deep neural networks and often fall into suboptimal solutions too close to the given dataset. In this work, we propose an uncertainty-based offline RL method that takes into account the confidence of the Q-value prediction and does not require any estimation or sampling of the data distribution. We show that the clipped Q-learning, a technique widely used in online RL, can be leveraged to successfully penalize OOD data points with high prediction uncertainties. Surprisingly, we find that it is possible to substantially outperform existing offline RL methods on various tasks by simply increasing the number of Q-networks along with the clipped Q-learning. Based on this observation, we propose an ensemble-diversified actor-critic algorithm that reduces the number of required ensemble networks down to a tenth compared to the naive ensemble while achieving state-of-the-art performance on most of the D4RL benchmarks considered.

Key facts. Cited by 297 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 20 of its 32 references that are in the map. First posted on arXiv 4 Oct 2021.

Builds on

20 of 32 references.

Closest ancestors · 7

Same direction, newest first

  1. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020MOPO: Model-based Offline Policy Optimization27 May
  4. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 13

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2016Deep Exploration via Bootstrapped DQNcited by 518
  7. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  8. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  9. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  10. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  11. 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 68
  12. 2018Generative Adversarial User Model for Reinforcement Learning Based Recommendation Systemcited by 64
1 more
  1. 2020Selective Dyna-style Planning Under Limited Model Capacitycited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 34; 2023: 64; 2024: 74; 2025: 72; 2026: 52.

297 papers.

Sort:

By area:

  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  2. 2025Flow Q-Learningcited by 162
  3. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  4. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  5. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  6. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  7. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  8. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  9. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  10. 2023Synthetic Experience Replaycited by 100
  11. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  12. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
38 more
  1. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  2. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  3. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  4. 2025Horizon Reduction Makes RL Scalablecited by 55
  5. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  6. 2023Anti-Exploration by Random Network Distillationcited by 49
  7. 2023Behavior Proximal Policy Optimizationcited by 43
  8. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  9. 2024Policy-Guided Diffusioncited by 40
  10. 2022A Review of Uncertainty for Deep Reinforcement Learningcited by 36
  11. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  12. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  13. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learningcited by 36
  14. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learningcited by 33
  15. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
  16. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
  17. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learningcited by 31
  18. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 31
  19. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 30
  20. 2024Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learningcited by 29
  21. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
  22. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  23. 2022Controlling Commercial Cooling Systems Using Reinforcement Learningcited by 25
  24. 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioningcited by 24
  25. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasetscited by 24
  26. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  27. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Beliefcited by 22
  28. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 21
  29. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  30. 2024Mixture of Experts in a Mixture of RL settingscited by 21
  31. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  32. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 20
  33. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learningcited by 20
  34. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
  35. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
  36. 2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
  37. 2023Supported Trust Region Optimization for Offline Reinforcement Learningcited by 20
  38. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18

50 most cited of 297 All 297 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Strategically Conservative Q-Learning
  2. 2021The Least Restriction for Offline Reinforcement Learning
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning
  4. 2021COMBO: Conservative Offline Model-Based Policy Optimization
  5. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  6. 2019An Optimistic Perspective on Offline Reinforcement Learning
  7. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learning
  8. 2025Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning