mlsanityData to 14 Oct 2026

Imagination-Limited Q-Learning for Offline Reinforcement Learning

arXiv 2505.12211 ↗First posted 18 May 2025cs.LG cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning seeks to derive improved policies entirely from historical data but often struggles with over-optimistic value estimates for out-of-distribution (OOD) actions. This issue is typically mitigated via policy constraint or conservative value regularization methods. However, these approaches may impose overly constraints or biased value estimates, potentially limiting performance improvements. To balance exploitation and restriction, we propose an Imagination-Limited Q-learning (ILQ) method, which aims to maintain the optimism that OOD actions deserve within appropriate limits. Specifically, we utilize the dynamics model to imagine OOD action-values, and then clip the imagined values with the maximum behavior values. Such design maintains reasonable evaluation of OOD actions to the furthest extent, while avoiding its over-optimism. Theoretically, we prove the convergence of the proposed ILQ under tabular Markov decision processes. Particularly, we demonstrate that the error bound between estimated values and optimality values of OOD state-actions possesses the same magnitude as that of in-distribution ones, thereby indicating that the bias in value estimates is effectively mitigated. Empirically, our method achieves state-of-the-art performance on a wide range of tasks in the D4RL benchmark.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 44 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

34 of 44 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024Offline Reinforcement Learning with Behavioral Supervisor Tuning25 Apr
  2. 2023Boosting Offline Reinforcement Learning with Action Preference Query6 Jun
  3. 2023Adversarial Model for Offline Reinforcement Learning21 Feb
  4. 2023Conservative State Value Estimation for Offline Reinforcement Learning14 Feb
  5. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  6. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning23 May
  7. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Offline Reinforcement Learning with Reverse Model-based Imagination1 Oct
  10. 2021Offline RL Without Off-Policy Evaluation16 Jun
  11. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  12. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
7 more
  1. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020MOPO: Model-based Offline Policy Optimization27 May
  4. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 15

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  9. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  10. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  12. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
3 more
  1. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 225
  2. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learningcited by 43
  3. 2021UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planningcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Offline Policy Optimization with Posterior Samplingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Strategically Conservative Q-Learning
  2. 2026In-Distribution Imagination for Model-Based Offline Reinforcement Learning
  3. 2024Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model
  4. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning
  5. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  6. 2026On Training in Imagination
  7. 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning
  8. 2024Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning