mlsanityData to 7 Oct 2026

Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model

arXiv 2410.20312 ↗First posted 27 Oct 2024cs.LG stat.MLcited by 4

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

``Distribution shift'' is the main obstacle to the success of offline reinforcement learning. A learning policy may take actions beyond the behavior policy's knowledge, referred to as Out-of-Distribution (OOD) actions. The Q-values for these OOD actions can be easily overestimated. As a result, the learning policy is biased by using incorrect Q-value estimates. One common approach to avoid Q-value overestimation is to make a pessimistic adjustment. Our key idea is to penalize the Q-values of OOD actions associated with high uncertainty. In this work, we propose Q-Distribution Guided Q-Learning (QDQ), which applies a pessimistic adjustment to Q-values in OOD regions based on uncertainty estimation. This uncertainty measure relies on the conditional Q-value distribution, learned through a high-fidelity and efficient consistency model. Additionally, to prevent overly conservative estimates, we introduce an uncertainty-aware optimization objective for updating the Q-value function. The proposed QDQ demonstrates solid theoretical guarantees for the accuracy of Q-value distribution learning and uncertainty measurement, as well as the performance of the learning policy. QDQ consistently shows strong performance on the D4RL benchmark and achieves significant improvements across many tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 57 of its 72 references that are in the map. First posted on arXiv 27 Oct 2024.

Builds on

57 of 72 references.

Closest ancestors · 24

Same direction, newest first

  1. 2023Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching24 Jun
  2. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  3. 2023The In-Sample Softmax for Offline Reinforcement Learning28 Feb
  4. 2023Conservative State Value Estimation for Offline Reinforcement Learning14 Feb
  5. 2023Constrained Policy Optimization with Explicit Behavior Density for Offline Reinforcement Learning28 Jan
  6. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  7. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian1 Nov
  8. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  9. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL1 Jun
  10. 2022Latent-Variable Advantage-Weighted Policy Optimization for Offline RL16 Mar
  11. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  12. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
12 more
  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  3. 2021Offline RL Without Off-Policy Evaluation16 Jun
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  6. 2021Risk-Averse Offline Reinforcement Learning10 Feb
  7. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  8. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  9. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  12. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 33

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  7. 2015Continuous control with deep reinforcement learningcited by 3,279
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  9. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  11. 2023Consistency Modelscited by 1,778
  12. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
21 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  3. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  4. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  10. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  12. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  13. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  14. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  15. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  16. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  17. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  18. 2017The Uncertainty Bellman Equation and Explorationcited by 81
  19. 2019Quantile QT-Opt for Risk-Aware Vision-Based Robotic Graspingcited by 27
  20. 2020Planning with Learned Dynamics: Probabilistic Guarantees on Safety and Reachability via Lipschitz Constantscited by 25
  21. 2023Uncertainty-Aware Instance Reweighting for Off-Policy Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  2. 2026Distributional Reinforcement Learning with Diffusion Bridge Criticscited by 0
  3. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learningcited by 0
  4. 2026Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Strategically Conservative Q-Learning
  2. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning
  3. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  4. 2024Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning
  5. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity
  6. 2026Uncertainty-Aware Rank-One MIMO Q Network Framework for Accelerated Offline Reinforcement Learning
  7. 2023Contextual Conservative Q-Learning for Offline Reinforcement Learning
  8. 2025Imagination-Limited Q-Learning for Offline Reinforcement Learning