mlsanityData to 13 Oct 2026

DIAR: Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation

arXiv 2410.11338 ↗First posted 15 Oct 2024cs.LG cs.AI cs.ROcited by 2

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 2 more
Abstract (arXiv)

We propose a novel offline reinforcement learning (offline RL) approach, introducing the Diffusion-model-guided Implicit Q-learning with Adaptive Revaluation (DIAR) framework. We address two key challenges in offline RL: out-of-distribution samples and long-horizon problems. We leverage diffusion models to learn state-action sequence distributions and incorporate value functions for more balanced and adaptive decision-making. DIAR introduces an Adaptive Revaluation mechanism that dynamically adjusts decision lengths by comparing current and future state values, enabling flexible long-term decision-making. Furthermore, we address Q-value overestimation by combining Q-network learning with a value function guided by a diffusion model. The diffusion model generates diverse latent trajectories, enhancing policy robustness and generalization. As demonstrated in tasks like Maze2D, AntMaze, and Kitchen, DIAR consistently outperforms state-of-the-art algorithms in long-horizon, sparse-reward environments.

Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 30 of its 32 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

30 of 32 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution18 Dec
  2. 2023Learning to Reach Goals via Diffusion4 Oct
  3. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learning12 Sep
  4. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  5. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  6. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 24

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,952
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,023
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,564
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,339
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,310
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,889
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,782
  9. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,710
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,531
  11. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,425
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,415
12 more
  1. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,257
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,231
  3. 2015Prioritized Experience Replaycited by 1,078
  4. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,035
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 731
  7. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  8. 2018Deep Reinforcement Learning and the Deadly Triadcited by 172
  9. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 165
  10. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  11. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  12. 2022Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discoverycited by 12

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Diffusion-based Planning with Learned Viability Filterscited by 2
  2. 2026Attention-DP3: Spatially Object-aware 3D Diffusion Policy via Geometry-aligned Attentional Conditioningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
  2. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  3. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning
  4. 2025DAWM: Diffusion Action World Models for Offline Reinforcement Learning via Action-Inferred Transitions
  5. 2026Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learning
  6. 2026Drift Q-Learning
  7. 2023Boosting Continuous Control with Consistency Policy
  8. 2025One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learning