mlsanityData to 14 Oct 2026

A Generalised Inverse Reinforcement Learning Framework

arXiv 2105.11812 ↗First posted 25 May 2021cs.LGcited by 2

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

The gloabal objective of inverse Reinforcement Learning (IRL) is to estimate the unknown cost function of some MDP base on observed trajectories generated by (approximate) optimal policies. The classical approach consists in tuning this cost function so that associated optimal trajectories (that minimise the cumulative discounted cost, i.e. the classical RL loss) are 'similar' to the observed ones. Prior contributions focused on penalising degenerate solutions and improving algorithmic scalability. Quite orthogonally to them, we question the pertinence of characterising optimality with respect to the cumulative discounted cost as it induces an implicit bias against policies with longer mixing times. State of the art value based RL algorithms circumvent this issue by solving for the fixed point of the Bellman optimality operator, a stronger criterion that is not well defined for the inverse problem. To alleviate this bias in IRL, we introduce an alternative training loss that puts more weights on future states which yields a reformulation of the (maximum entropy) IRL problem. The algorithms we devised exhibit enhanced performances (and similar tractability) than off-the-shelf ones in multiple OpenAI gym environments.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 15 of its 41 references that are in the map. First posted on arXiv 25 May 2021.

Builds on

15 of 41 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Regularized Inverse Reinforcement Learning7 Oct
  2. 2018Adversarial Imitation via Variational Inverse Reinforcement Learning17 Sep

Other references · 13

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2014Conditional Generative Adversarial Netscited by 2,162
  5. 2016Generative Adversarial Imitation Learningcited by 1,597
  6. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  8. 2019A Theory of Regularized Markov Decision Processescited by 286
  9. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  10. 2019Ray Interference: a Source of Plateaus in Deep Reinforcement Learningcited by 40
  11. 2016Learning to Drive using Inverse Reinforcement Learning and Deep Q-Networkscited by 15
  12. 2019Situated GAIL: Multitask imitation using task-conditioned adversarial inverse reinforcement learningcited by 4
1 more
  1. 2019Markov Decision Process for MOOC users behavioral inferencecited by 2

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 0; 2023: 1.

2 papers.

  1. 2021Offline Inverse Reinforcement Learningcited by 14
  2. 2023On Combining Expert Demonstrations in Imitation Learning via Optimal Transportcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  2. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  3. 2026Fast Rates for Inverse Reinforcement Learning
  4. 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review
  5. 2026A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models
  6. 2026Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching
  7. 2025Inverse Reinforcement Learning via Convex Optimization
  8. 2025Cost Function Estimation Using Inverse Reinforcement Learning with Minimal Observations