mlsanityData to 7 Oct 2026

UDQL: Bridging The Gap between MSE Loss and The Optimal Value Function in Offline Reinforcement Learning

arXiv 2406.03324 ↗First posted 5 Jun 2024cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The Mean Square Error (MSE) is commonly utilized to estimate the solution of the optimal value function in the vast majority of offline reinforcement learning (RL) models and has achieved outstanding performance. However, we find that its principle can lead to overestimation phenomenon for the value function. In this paper, we first theoretically analyze overestimation phenomenon led by MSE and provide the theoretical upper bound of the overestimated error. Furthermore, to address it, we propose a novel Bellman underestimated operator to counteract overestimation phenomenon and then prove its contraction characteristics. At last, we propose the offline RL algorithm based on underestimated operator and diffusion policy model. Extensive experimental results on D4RL tasks show that our method can outperform state-of-the-art offline RL algorithms, which demonstrates that our theoretical analysis and underestimation way are effective for offline RL tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 46 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

34 of 46 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  2. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  3. 2022Mutual Information Regularized Offline Reinforcement Learning14 Oct
  4. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  5. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning23 May
  6. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  7. 2021Offline Reinforcement Learning with Soft Behavior Regularization14 Oct
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Offline RL Without Off-Policy Evaluation16 Jun
  10. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  11. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  12. 2020Critic Regularized Regression26 Jun
3 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  3. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb

Other references · 19

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  8. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  9. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  10. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  11. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  12. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
7 more
  1. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
  2. 2019Mish: A Self Regularized Non-Monotonic Activation Functioncited by 141
  3. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  4. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 90
  5. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  6. 2023Score Regularized Policy Optimization through Diffusion Behaviorcited by 67
  7. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Oracle Inequalities for Model Selection in Offline Reinforcement Learning
  2. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning
  3. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  4. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning
  5. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game
  6. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning
  7. 2023Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning
  8. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation