mlsanityData to 7 Oct 2026

IQ-Learn: Inverse soft-Q Learning for Imitation

arXiv 2106.12142 ↗First posted 23 Jun 2021cs.LG cs.AIcited by 191

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

In many sequential decision-making problems (e.g., robotics control, game playing, sequential prediction), human or expert data is available containing useful information about the task. However, imitation learning (IL) from a small amount of expert data can be challenging in high-dimensional environments with complex dynamics. Behavioral cloning is a simple method that is widely used due to its simplicity of implementation and stable convergence but doesn't utilize any information involving the environment's dynamics. Many existing methods that exploit dynamics information are difficult to train in practice due to an adversarial optimization process over reward and policy approximators or biased, high variance gradient estimators. We introduce a method for dynamics-aware IL which avoids adversarial training by learning a single Q-function, implicitly representing both reward and policy. On standard benchmarks, the implicitly learned rewards show a high positive correlation with the ground-truth rewards, illustrating our method can also be used for inverse reinforcement learning (IRL). Our method, Inverse soft-Q learning (IQ-Learn) obtains state-of-the-art results in offline and online imitation learning settings, significantly outperforming existing methods both in the number of required environment interactions and scalability in high-dimensional spaces, often by more than 3x.

Key facts. Cited by 191 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 20 of its 44 references that are in the map. First posted on arXiv 23 Jun 2021.

Builds on

20 of 44 references.

Closest ancestors · 6

Same direction, newest first

  1. 2020Non-Adversarial Imitation Learning and its Connections to Adversarial Methods8 Aug
  2. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matching25 Jun
  3. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization23 Jun
  4. 2020Primal Wasserstein Imitation Learning8 Jun
  5. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  6. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep

Other references · 14

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2017Improved Training of Wasserstein GANscited by 2,223
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  8. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  9. 2017Are GANs Created Equal? A Large-Scale Studycited by 267
  10. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variablescited by 50
  11. 2016Model-based Adversarial Imitation Learningcited by 29
  12. 2019Truly Batch Apprenticeship Learning with Deep Successor Featurescited by 27
2 more
  1. 2016Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamicscited by 24
  2. 2021Exploring Imitation Learning for Autonomous Driving with Feedback Synthesizer and Differentiable Rasterizationcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 22; 2023: 44; 2024: 58; 2025: 41; 2026: 26.

191 papers.

Sort:

By area:

  1. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  2. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  3. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  4. 2023SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Executioncited by 69
  5. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  6. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  7. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  8. 2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guaranteescited by 42
  9. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  10. 2023Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RLcited by 37
  11. 2023Inverse Reinforcement Learning without Reinforcement Learningcited by 35
  12. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
38 more
  1. 2023LocoMuJoCo: A Comprehensive Imitation Learning Benchmark for Locomotioncited by 31
  2. 2024Hybrid Inverse Reinforcement Learningcited by 31
  3. 2022LISA: Learning Interpretable Skill Abstractions from Languagecited by 30
  4. 2022Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrationscited by 30
  5. 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learningcited by 28
  6. 2023When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learningcited by 26
  7. 2022Proximal Point Imitation Learningcited by 25
  8. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 22
  9. 2023Coherent Soft Imitation Learningcited by 21
  10. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
  11. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  12. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  13. 2022Target-absent Human Attentioncited by 20
  14. 2023CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learningcited by 20
  15. 2022A Ranking Game for Imitation Learningcited by 18
  16. 2022Benchmarking Constraint Inference in Inverse Reinforcement Learningcited by 18
  17. 2023DiffAIL: Diffusion Adversarial Imitation Learningcited by 18
  18. 2024Diffusion-Reward Adversarial Imitation Learningcited by 18
  19. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
  20. 2025Implicit Reward as the Bridge: A Unified View of SFT and DPO Connectionscited by 17
  21. 2023Visual Imitation Learning with Patch Rewardscited by 16
  22. 2023Identifiability and Generalizability in Constrained Inverse Reinforcement Learningcited by 16
  23. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitionscited by 16
  24. 2023B-Coder: Value-Based Deep Reinforcement Learning for Program Synthesiscited by 16
  25. 2024Diffusing States and Matching Scores: A New Framework for Imitation Learningcited by 16
  26. 2023Hierarchical Imitation Learning with Vector Quantized Modelscited by 15
  27. 2023Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competitioncited by 15
  28. 2024Imitation Learning in Discounted Linear MDPs without exploration assumptionscited by 15
  29. 2023SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtrackingcited by 13
  30. 2025On a Connection Between Imitation Learning and RLHFcited by 13
  31. 2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guaranteescited by 12
  32. 2023CEIL: Generalized Contextual Imitation Learningcited by 12
  33. 2024Diffusion Imitation from Observationcited by 12
  34. 2024Learning to Assist Humans without Inferring Rewardscited by 12
  35. 2023Massively Scalable Inverse Reinforcement Learning in Google Mapscited by 11
  36. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
  37. 2024Inverse Reinforcement Learning by Estimating Expertise of Demonstratorscited by 11
  38. 2024Efficient Imitation Learning with Conservative World Modelscited by 11

50 most cited of 191 All 191 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Extrinsicaly Rewarded Soft Q Imitation Learning with Discriminator
  2. 2019A Divergence Minimization Perspective on Imitation Learning Methods
  3. 2023Aligning Human Intent from Imperfect Demonstrations with Confidence-based Inverse soft-Q Learning
  4. 2020Discriminator Soft Actor Critic without Extrinsic Rewards
  5. 2023Robust Visual Imitation Learning with Inverse Dynamics Representations
  6. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
  7. 2019Imitation Learning as f-Divergence Minimization
  8. 2025Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advances