mlsanityData to 7 Oct 2026

Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning

arXiv 2404.06188 ↗First posted 9 Apr 2024cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Reinforcement Learning (RL) faces distributional shift and unreliable value estimation, especially for out-of-distribution (OOD) actions. To address this, existing uncertainty-based methods penalize the value function with uncertainty quantification and demand numerous ensemble networks, posing computational challenges and suboptimal outcomes. In this paper, we introduce a novel strategy employing diverse randomized value functions to estimate the posterior distribution of Q-values. It provides robust uncertainty quantification and estimates lower confidence bounds (LCB) of Q-values. By applying moderate value penalties for OOD actions, our method fosters a provably pessimistic approach. We also emphasize on diversity within randomized value functions and enhance efficiency by introducing a diversity regularization method, reducing the requisite number of networks. These modules lead to reliable value estimation and efficient policy learning from offline data. Theoretical analysis shows that our method recovers the provably efficient LCB-penalty under linear MDP assumptions. Extensive empirical results also demonstrate that our proposed method significantly outperforms baseline methods in terms of performance and parametric efficiency.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 49 references that are in the map. First posted on arXiv 9 Apr 2024.

Builds on

43 of 49 references.

Closest ancestors · 17

Same direction, newest first

  1. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  2. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  3. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  4. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  7. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
5 more
  1. 2020MOPO: Model-based Offline Policy Optimization27 May
  2. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 26

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  9. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  10. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  11. 2016Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithmcited by 498
  12. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
14 more
  1. 2017Noisy Networks for Explorationcited by 262
  2. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  3. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  4. 2017Deep Exploration via Randomized Value Functionscited by 155
  5. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  6. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 149
  7. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  8. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 108
  9. 2021Repulsive Deep Ensembles are Bayesiancited by 103
  10. 2018Uncertainty in Neural Networks: Approximately Bayesian Ensemblingcited by 94
  11. 2022Agree to Disagree: Diversity through Disagreement for Better Transferabilitycited by 69
  12. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  13. 2020Depth Uncertainty in Neural Networkscited by 56
  14. 2022Bootstrapped Transformer for Offline Reinforcement Learningcited by 37

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Epistemic Robust Offline Reinforcement Learning
  2. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning
  3. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning
  4. 2024Strategically Conservative Q-Learning
  5. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism
  6. 2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning
  7. 2024Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model
  8. 2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approach