Data to 7 Oct 2026

IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies

arXiv 2304.10573 ↗First posted 20 Apr 2023cs.LG cs.AIcited by 307

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more

Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Effective offline RL methods require properly handling out-of-distribution actions. Implicit Q-learning (IQL) addresses this by training a Q-function using only dataset actions through a modified Bellman backup. However, it is unclear which policy actually attains the values represented by this implicitly trained Q-function. In this paper, we reinterpret IQL as an actor-critic method by generalizing the critic objective and connecting it to a behavior-regularized implicit actor. This generalization shows how the induced actor balances reward maximization and divergence from the behavior policy, with the specific loss choice determining the nature of this tradeoff. Notably, this actor can exhibit complex and multimodal characteristics, suggesting issues with the conditional Gaussian actor fit with advantage weighted regression (AWR) used in prior methods. Instead, we propose using samples from a diffusion parameterized behavior policy and weights computed from the critic to then importance sampled our intended policy. We introduce Implicit Diffusion Q-learning (IDQL), combining our general IQL critic with the policy extraction method. IDQL maintains the ease of implementation of IQL while outperforming prior offline RL methods and demonstrating robustness to hyperparameters. Code is available at https://github.com/philippe-eecs/IDQL.

Key facts. Cited by 307 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 41 of its 51 references that are in the map. First posted on arXiv 20 Apr 2023.

Builds on

41 of 51 references.

Closest ancestors · 4

Same direction, newest first

Other references · 37

2015Deep Residual Learning for Image Recognitioncited by 51,809
2014Adam: A Method for Stochastic Optimizationcited by 36,960
2020Denoising Diffusion Probabilistic Modelscited by 20,851
2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
2016Layer Normalizationcited by 5,579
2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528

25 more

2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
2019Behavior Regularized Offline Reinforcement Learningcited by 654
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
2021Implicit Behavioral Cloningcited by 442
2023Efficient Online Reinforcement Learning with Offline Datacited by 332
2020Critic Regularized Regressioncited by 269
2018What is the Effect of Importance Weighting in Deep Learning?cited by 250
2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
2021Offline RL Without Off-Policy Evaluationcited by 150
2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 113
2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
2021Understanding the role of importance weighting for deep learningcited by 38
2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RLcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 69; 2025: 105; 2026: 109.

307 papers.

Sort:

By area:

2023Training Diffusion Models with Reinforcement Learningcited by 939
2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
2024Diffusion Policy Policy Optimizationcited by 255
2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 234
2025Flow Q-Learningcited by 162
2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 126
2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 99
2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93
2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 92

38 more

2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
2024One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillationcited by 83
2025ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learningcited by 83
2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 72
2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
2024Amortizing intractable inference in diffusion models for vision, language, and controlcited by 70
2025Energy-Weighted Flow Matching for Offline Reinforcement Learningcited by 70
2023Score Regularized Policy Optimization through Diffusion Behaviorcited by 67
2023Elastic Decision Transformercited by 66
2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
2025Efficient Online Reinforcement Learning for Diffusion Policycited by 61
2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
2024Unveil Conditional Diffusion Models with Classifier-free Guidance: A Sharp Statistical Theorycited by 58
2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 56
2025Horizon Reduction Makes RL Scalablecited by 55
2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 52
2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
2024AdaFlow: Imitation Learning with Variance-Adaptive Flow-Based Policiescited by 50
2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 49
2023PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Playcited by 48
2024Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Samplingcited by 46
2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learningcited by 41
2024CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Makingcited by 38
2025What Makes a Good Diffusion Planner for Decision Making?cited by 37
2023Boosting Continuous Control with Consistency Policycited by 36
2024Stochastic Runge-Kutta Methods: Provable Acceleration of Diffusion Modelscited by 35
2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
2023World Models via Policy-Guided Trajectory Diffusioncited by 34
2023Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behaviorcited by 33
2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
2026Q-learning with Adjoint Matchingcited by 33
2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 31
2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30

50 most cited of 307 All 307 (JSON)

Similar, not cited

Close in wording, no citation either way