mlsanityData to 14 Oct 2026

Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shaping

arXiv 2410.03847 ↗First posted 4 Oct 2024cs.LG cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

also in 1 more
Abstract (arXiv)

In this paper, we aim to tackle the limitation of the Adversarial Inverse Reinforcement Learning (AIRL) method in stochastic environments where theoretical results cannot hold and performance is degraded. To address this issue, we propose a novel method which infuses the dynamics information into the reward shaping with the theoretical guarantee for the induced optimal policy in the stochastic environments. Incorporating our novel model-enhanced rewards, we present a novel Model-Enhanced AIRL framework, which integrates transition model estimation directly into reward shaping. Furthermore, we provide a comprehensive theoretical analysis of the reward error bound and performance difference bound for our method. The experimental results in MuJoCo benchmarks show that our method can achieve superior performance in stochastic environments and competitive performance in deterministic environments, with significant improvement in sample efficiency, compared to existing baselines.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 52 of its 84 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

52 of 84 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021Scalable Bayesian Inverse Reinforcement Learning12 Feb
  2. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch2 Jul
  3. 2018Inverse reinforcement learning for video games24 Oct
  4. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress18 Jun

Other references · 48

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  4. 2016Generative Adversarial Imitation Learningcited by 1,597
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  7. 2020Mastering Atari with Discrete World Modelscited by 1,007
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  9. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  10. 2019Model-Based Reinforcement Learning for Ataricited by 502
  11. 2018Behavioral Cloning from Observationcited by 492
  12. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
36 more
  1. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  2. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  3. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  4. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  5. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 251
  6. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 236
  7. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 204
  8. 2018Generative Adversarial Imitation from Observationcited by 184
  9. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 176
  10. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 168
  11. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  12. 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 114
  13. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 81
  14. 2021What Matters for Adversarial Imitation Learning?cited by 70
  15. 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 62
  16. 2022Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environmentscited by 60
  17. 2020Learning Off-Policy with Online Planningcited by 53
  18. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrationscited by 52
  19. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  20. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  21. 2021Self-Supervised Online Reward Shaping in Sparse-Reward Environmentscited by 37
  22. 2017Data-driven Planning via Imitation Learningcited by 36
  23. 2018Sample-Efficient Imitation Learning via Generative Adversarial Netscited by 36
  24. 2022imitation: Clean Imitation Learning Implementationscited by 34
  25. 2023Empowering Autonomous Driving with Large Language Models: A Safety Perspectivecited by 34
  26. 2020Model-based Adversarial Meta-Reinforcement Learningcited by 33
  27. 2016Model-based Adversarial Imitation Learningcited by 29
  28. 2016Inverse Reinforcement Learning with Simultaneous Estimation of Rewards and Dynamicscited by 24
  29. 2017A Fast Integrated Planning and Control Framework for Autonomous Driving via Imitation Learningcited by 20
  30. 2022Planning for Sample Efficient Imitation Learningcited by 20
  31. 2020Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learningcited by 15
  32. 2022Joint Differentiable Optimization and Verification for Certified Reinforcement Learningcited by 11
  33. 2024Boosting Reinforcement Learning with Strongly Delayed Feedback Through Auxiliary Short Delayscited by 10
  34. 2021A Pragmatic Look at Deep Imitation Learningcited by 9
  35. 2023State-Wise Safe Reinforcement Learning With Pixel Observationscited by 9
  36. 2024Variational Delayed Policy Optimizationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2024Inverse Delayed Reinforcement Learningcited by 0
  2. 2025Distributional Inverse Reinforcement Learningcited by 0
  3. 2026A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally Extended Actions
  2. 2019Multi-Agent Adversarial Inverse Reinforcement Learning
  3. 2025Hybrid-AIRL: Enhancing Inverse Reinforcement Learning with Supervised Expert Guidance
  4. 2025AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
  5. 2024On Reward Transferability in Adversarial Inverse Reinforcement Learning: Insights from Random Matrix Theory
  6. 2024Rethinking Adversarial Inverse Reinforcement Learning: Policy Imitation, Transferable Reward Recovery and Algebraic Equilibrium Proof
  7. 2025Offline Reinforcement Learning via Inverse Optimization
  8. 2020Off-Policy Adversarial Inverse Reinforcement Learning