mlsanityData to 7 Oct 2026

AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimization

arXiv 2405.18187 ↗First posted 28 May 2024cs.LGcited by 19

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Implicit Q-learning (IQL) serves as a strong baseline for offline RL, which learns the value function using only dataset actions through quantile regression. However, it is unclear how to recover the implicit policy from the learned implicit Q-function and why IQL can utilize weighted regression for policy extraction. IDQL reinterprets IQL as an actor-critic method and gets weights of implicit policy, however, this weight only holds for the optimal value function. In this work, we introduce a different way to solve the implicit policy-finding problem (IPF) by formulating this problem as an optimization problem. Based on this optimization problem, we further propose two practical algorithms AlignIQL and AlignIQL-hard, which inherit the advantages of decoupling actor from critic in IQL and provide insights into why IQL can use weighted regression for policy extraction. Compared with IQL and IDQL, we find our method keeps the simplicity of IQL and solves the implicit policy-finding problem. Experimental results on D4RL datasets show that our method achieves competitive or superior results compared with other SOTA offline RL methods. Especially in complex sparse reward tasks like Antmaze and Adroit, our method outperforms IQL and IDQL by a significant margin.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 58 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

46 of 58 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption19 Oct
  2. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  3. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  4. 2021Offline Reinforcement Learning with Value-based Episodic Memory19 Oct
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021Offline RL Without Off-Policy Evaluation16 Jun
  7. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  8. 2020Critic Regularized Regression26 Jun
  9. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 35

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  9. 2015Continuous control with deep reinforcement learningcited by 3,279
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  11. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  12. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
23 more
  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  3. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  7. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  8. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  9. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 373
  10. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  11. 2023Imitating Human Behaviour with Diffusion Modelscited by 260
  12. 2024Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generationcited by 211
  13. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  14. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  15. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  16. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
  17. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
  18. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  19. 2018Deep Imitative Models for Flexible Inference, Planning, and Controlcited by 75
  20. 2023Score Regularized Policy Optimization through Diffusion Behaviorcited by 67
  21. 2019Learning When-to-Treat Policiescited by 49
  22. 2023DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learningcited by 27
  23. 2024Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 8.

19 papers.

Sort:

By area:

  1. 2025Flow Q-Learningcited by 162
  2. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  3. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 66
  4. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  5. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
  6. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  7. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  8. 2025Unleashing Flow Policies with Distributional Criticscited by 7
  9. 2025Towards Robust Zero-Shot Reinforcement Learningcited by 7
  10. 2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
  11. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 4
  12. 2025Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruptioncited by 3
7 more
  1. 2025Consistency Trajectory Planning: High-Quality and Efficient Trajectory Optimization for Offline Model-Based Reinforcement Learningcited by 2
  2. 2026Reversal Q-Learningcited by 2
  3. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learningcited by 0
  4. 2026ACSAC: Adaptive Chunk Size Actor-Critic with Causal Transformer Q-Networkcited by 0
  5. 2026Aligning Flow Map Policies with Optimal Q-Guidancecited by 0
  6. 2026Shortcut Trajectory Planning for Efficient Offline Reinforcement Learningcited by 0
  7. 2026HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PIQL: Projective Implicit Q-Learning with Support Constraint for Offline Reinforcement Learning
  2. 2026Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment
  3. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  4. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone
  5. 2023IQL-TD-MPC: Implicit Q-Learning for Hierarchical Model Predictive Control
  6. 2025Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression
  7. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  8. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Training