mlsanityData to 14 Oct 2026

LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learning

arXiv 2303.00599 ↗First posted 1 Mar 2023cs.LG cs.AI cs.ROcited by 29

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Recent methods for imitation learning directly learn a Q-function using an implicit reward formulation rather than an explicit reward function. However, these methods generally require implicit reward regularization to improve stability and often mistreat absorbing states. Previous works show that a squared norm regularization on the implicit reward function is effective, but do not provide a theoretical analysis of the resulting properties of the algorithms. In this work, we show that using this regularizer under a mixture distribution of the policy and the expert provides a particularly illuminating perspective: the original objective can be understood as squared Bellman error minimization, and the corresponding optimization problem minimizes a bounded chi2-Divergence between the expert and the mixture distribution. This perspective allows us to address instabilities and properly treat absorbing states. We show that our method, Least Squares Inverse Q-Learning (LS-IQ), outperforms state-of-the-art algorithms, particularly in environments with absorbing states. Finally, we propose to use an inverse dynamics model to learn from observations only. Using this approach, we retain performance in settings where no expert actions are available.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 24 of its 35 references that are in the map. First posted on arXiv 1 Mar 2023.

Builds on

24 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019RIDM: Reinforced Inverse Dynamics Modeling for Learning from a Single Observed Demonstration18 Jun

Other references · 23

  1. 2022Generative Adversarial Networkscited by 5,627
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2016Generative Adversarial Imitation Learningcited by 1,597
  4. 2016Least Squares Generative Adversarial Networkscited by 1,001
  5. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
  6. 2018Behavioral Cloning from Observationcited by 492
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  8. 2019Deep Evidential Regressioncited by 355
  9. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  10. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 204
  11. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  12. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 190
11 more
  1. 2018Generative Adversarial Imitation from Observationcited by 184
  2. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 183
  3. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flowcited by 105
  4. 2019Recent Advances in Imitation Learning from Observationcited by 93
  5. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 91
  6. 2017Combining Self-Supervised Learning and Imitation for Vision-Based Rope Manipulationcited by 90
  7. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 81
  8. 2020MushroomRL: Simplifying Reinforcement Learning Researchcited by 50
  9. 2022A Ranking Game for Imitation Learningcited by 18
  10. 2020Non-Adversarial Imitation Learning and its Connections to Adversarial Methodscited by 17
  11. 2019Hybrid Reinforcement Learning with Expert State Sequencescited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 9; 2025: 6; 2026: 7.

29 papers.

Sort:

By area:

  1. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  2. 2023LocoMuJoCo: A Comprehensive Imitation Learning Benchmark for Locomotioncited by 33
  3. 2023Coherent Soft Imitation Learningcited by 22
  4. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
  5. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  6. 2024Diffusing States and Matching Scores: A New Framework for Imitation Learningcited by 16
  7. 2023SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtrackingcited by 13
  8. 2024A Dual Approach to Imitation Learning from Observations with Offline Datasetscited by 10
  9. 2023Learning Robot Manipulation from Cross-Morphology Demonstrationcited by 8
  10. 2024Reward-free World Models for Online Imitation Learningcited by 7
  11. 2024OLLIE: Imitation Learning from Offline Pretraining to Online Finetuningcited by 6
  12. 2025RAPID: Robust and Agile Planner Using Inverse Reinforcement Learning for Vision-Based Drone Navigationcited by 6
17 more
  1. 2023Mimicking Better by Matching the Approximate Action Distributioncited by 5
  2. 2024Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learningcited by 2
  3. 2025Language Model Distillation: A Temporal Difference Imitation Learning Perspectivecited by 2
  4. 2024Exciting Action: Investigating Efficient Exploration for Learning Musculoskeletal Humanoid Locomotioncited by 1
  5. 2024The Role of Domain Randomization in Training Diffusion Policies for Whole-Body Humanoid Controlcited by 1
  6. 2026Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learningcited by 1
  7. 2024UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrationscited by 0
  8. 2025RIZE: Adaptive Regularization for Imitation Learningcited by 0
  9. 2025DO-IQS: Dynamics-Aware Offline Inverse Q-Learning for Optimal Stopping with Unknown Gain Functionscited by 0
  10. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrationscited by 0
  11. 2025Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvementcited by 0
  12. 2026Latent Wasserstein Adversarial Imitation Learningcited by 0
  13. 2026Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Value Flow Mechanismcited by 0
  14. 2026Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updatescited by 0
  15. 2026Provable Benefits of Regularization: Fast Rates for Adversarial Imitation Learningcited by 0
  16. 2026Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matchingcited by 0
  17. 2026A Geometric Perspective on Reward Function Updates in Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Statistical analysis of Inverse Entropy-regularized Reinforcement Learning
  2. 2020Regularized Inverse Reinforcement Learning
  3. 2024Learning Causally Invariant Reward Functions from Diverse Demonstrations
  4. 2023A Novel Variational Lower Bound for Inverse Reinforcement Learning
  5. 2026Mixing Regularization for Average-Reward Q-Learning and Q-Planning
  6. 2026Decoding Rewards in Competitive Games: Inverse Game Theory with Entropy Regularization
  7. 2021Identifiability in inverse reinforcement learning
  8. 2025Inverse Reinforcement Learning with Just Classification and a Few Regressions