mlsanityData to 7 Oct 2026

Confidence-Conditioned Value Functions for Offline Reinforcement Learning

arXiv 2212.04607 ↗First posted 8 Dec 2022cs.LG cs.AIcited by 20

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) promises the ability to learn effective policies solely using existing, static datasets, without any costly online interaction. To do so, offline RL methods must handle distributional shift between the dataset and the learned policy. The most common approach is to learn conservative, or lower-bound, value functions, which underestimate the return of out-of-distribution (OOD) actions. However, such methods exhibit one notable drawback: policies optimized on such value functions can only behave according to a fixed, possibly suboptimal, degree of conservatism. However, this can be alleviated if we instead are able to learn policies for varying degrees of conservatism at training time and devise a method to dynamically choose one of them during evaluation. To do so, in this work, we propose learning value functions that additionally condition on the degree of conservatism, which we dub confidence-conditioned value functions. We derive a new form of a Bellman backup that simultaneously learns Q-values for any degree of confidence with high probability. By conditioning on confidence, our value functions enable adaptive strategies during online evaluation by controlling for confidence level using the history of observations thus far. This approach can be implemented in practice by conditioning the Q-function from existing conservative algorithms on the confidence.We theoretically show that our learned value functions produce conservative estimates of the true value at any desired confidence. Finally, we empirically show that our algorithm outperforms existing conservative offline RL algorithms on multiple discrete control domains.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 45 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

36 of 45 references.

Closest ancestors · 16

Same direction, newest first

  1. 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
  2. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularization9 Dec
  3. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  4. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  5. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  6. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  7. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  8. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  9. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  10. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020MOPO: Model-based Offline Policy Optimization27 May
4 more
  1. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 20

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2018Exploration by Random Network Distillationcited by 966
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  8. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  10. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  11. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  12. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
8 more
  1. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  2. 2021Reinforcement learning based recommender systems: A surveycited by 187
  3. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  4. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  5. 2017The Uncertainty Bellman Equation and Explorationcited by 81
  6. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  7. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  8. 2020Minimax Value Interval for Off-Policy Evaluation and Policy Optimizationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 7; 2025: 1; 2026: 2.

20 papers.

Sort:

By area:

  1. 2023Anti-Exploration by Random Network Distillationcited by 49
  2. 2024Exploration and Anti-Exploration with Distributional Random Network Distillationcited by 32
  3. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  4. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  5. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
  6. 2023Model-based Offline Reinforcement Learning with Count-based Conservatismcited by 16
  7. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 10
  8. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  9. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Biascited by 7
  10. 2024An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learningcited by 5
  11. 2023Budgeting Counterfactual for Offline RLcited by 4
  12. 2023Learning Control Policies for Variable Objectives from Offline Datacited by 3
8 more
  1. 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flowscited by 2
  2. 2024Preference-Optimized Pareto Set Learning for Blackbox Optimizationcited by 2
  3. 2026Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weightingcited by 2
  4. 2024Grid-Mapping Pseudo-Count Constraint for Offline Reinforcement Learningcited by 1
  5. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  6. 2026Beyond Penalization: Diffusion-based Out-of-Distribution Detection and Selective Regularization in Offline Reinforcement Learningcited by 1
  7. 2023Automatic Trade-off Adaptation in Offline RLcited by 0
  8. 2023Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Explorationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Strategically Conservative Q-Learning
  2. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  3. 2024Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning
  4. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  5. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning
  7. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning
  8. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization