mlsanityData to 7 Oct 2026

Offline Reinforcement Learning with Implicit Q-Learning

arXiv 2110.06169 ↗First posted 12 Oct 2021cs.LGcited by 1,245

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning requires reconciling two conflicting aims: learning a policy that improves over the behavior policy that collected the dataset, while at the same time minimizing the deviation from the behavior policy so as to avoid errors due to distributional shift. This trade-off is critical, because most current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy, and therefore need to either constrain these actions to be in-distribution, or else regularize their values. We propose an offline RL method that never needs to evaluate actions outside of the dataset, but still enables the learned policy to improve substantially over the best behavior in the data through generalization. The main insight in our work is that, instead of evaluating unseen actions from the latest policy, we can approximate the policy improvement step implicitly by treating the state value function as a random variable, with randomness determined by the action (while still integrating over the dynamics to avoid excessive optimism), and then taking a state conditional upper expectile of this random variable to estimate the value of the best actions in that state. This leverages the generalization capacity of the function approximator to estimate the value of the best available action at a given state without ever directly querying a Q-function with this unseen action. Our algorithm alternates between fitting this upper expectile value function and backing it up into a Q-function. Then, we extract the policy via advantage-weighted behavioral cloning. We dub our method implicit Q-learning (IQL). IQL demonstrates the state-of-the-art performance on D4RL, a standard benchmark for offline reinforcement learning. We also demonstrate that IQL achieves strong performance fine-tuning using online interaction after offline initialization.

Key facts. Cited by 1,245 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 23 of its 35 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

23 of 35 references.

Closest ancestors · 11

Same direction, newest first

  1. 2021Offline RL Without Off-Policy Evaluation16 Jun
  2. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  3. 2021Regularized Behavior Value Estimation17 Mar
  4. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  5. 2021Continuous Doubly Constrained Batch Reinforcement Learning18 Feb
  6. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  7. 2020Critic Regularized Regression26 Jun
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  11. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 12

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  9. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  10. 2021Implicit Behavioral Cloningcited by 442
  11. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  12. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 92; 2023: 215; 2024: 258; 2025: 310; 2026: 364.

1,245 papers.

Sort:

By area:

  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  2. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  3. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  4. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  5. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  6. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  7. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  8. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  9. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  10. 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixelscited by 237
  11. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  12. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
38 more
  1. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  2. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  3. 2022Online Decision Transformercited by 194
  4. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  5. 2025Flow Q-Learningcited by 162
  6. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  7. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  8. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  9. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
  10. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  11. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
  12. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  13. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  14. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 132
  15. 2023Large Language Models for Robotics: A Surveycited by 132
  16. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  17. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  18. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  19. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  20. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  21. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  22. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 113
  23. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  24. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  25. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  26. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  27. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  28. 2022Jump-Start Reinforcement Learningcited by 102
  29. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  30. 2023Synthetic Experience Replaycited by 100
  31. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  32. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  33. 2025Reinforcement Learning with Action Chunkingcited by 97
  34. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 93
  35. 2024Foundation Policies with Hilbert Representationscited by 89
  36. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  37. 2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
  38. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84

50 most cited of 1,245 All 1,245 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble
  2. 2023Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations
  3. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  4. 2021The Least Restriction for Offline Reinforcement Learning
  5. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning
  6. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?
  7. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
  8. 2023Offline Reinforcement Learning with Additional Covering Distributions