mlsanityData to 7 Oct 2026

Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms

arXiv 2012.00889 ↗First posted 1 Dec 2020cs.LG cs.ROcited by 10

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

We provide new perspectives and inference algorithms for Maximum Entropy (MaxEnt) Inverse Reinforcement Learning (IRL), which provides a principled method to find a most non-committal reward function consistent with given expert demonstrations, among many consistent reward functions. We first present a generalized MaxEnt formulation based on minimizing a KL-divergence instead of maximizing an entropy. This improves the previous heuristic derivation of the MaxEnt IRL model (for stochastic MDPs), allows a unified view of MaxEnt IRL and Relative Entropy IRL, and leads to a model-free learning algorithm for the MaxEnt IRL model. Second, a careful review of existing inference algorithms and implementations showed that they approximately compute the marginals required for learning the model. We provide examples to illustrate this, and present an efficient and exact inference algorithm. Our algorithm can handle variable length demonstrations; in addition, while a basic version takes time quadratic in the maximum demonstration length L, an improved version of this algorithm reduces this to linear using a padding trick. Experiments show that our exact algorithm improves reward learning as compared to the approximate ones. Furthermore, our algorithm scales up to a large, real-world dataset involving driver behaviour forecasting. We provide an optimized implementation compatible with the OpenAI Gym interface. Our new insight and algorithms could possibly lead to further interest and exploration of the original MaxEnt IRL model.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 4 of its 23 references that are in the map. First posted on arXiv 1 Dec 2020.

Builds on

4 of 23 references.

References · 4

  1. 2016OpenAI Gymcited by 1,684
  2. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  3. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  4. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 167

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 1; 2023: 0; 2024: 3; 2025: 3; 2026: 2.

10 papers.

  1. 2025Confucius3-Math: A Lightweight High-Performance Reasoning LLM for Chinese K-12 Mathematics Learningcited by 5
  2. 2022X-MEN: Guaranteed XOR-Maximum Entropy Constrained Inverse Reinforcement Learningcited by 4
  3. 2024Maximum Causal Entropy IRL in Mean-Field Games and GNEP Framework for Forward RLcited by 4
  4. 2025Inverse Reinforcement Learning with Just Classification and a Few Regressionscited by 4
  5. 2021LiMIIRL: Lightweight Multiple-Intent Inverse Reinforcement Learningcited by 2
  6. 2024Inverse Reinforcement Learning from Non-Stationary Learning Agentscited by 1
  7. 2026Can Optimal Transport Improve Federated Inverse Reinforcement Learning?cited by 1
  8. 2024Residual Chain Prediction for Autonomous Driving Path Planningcited by 0
  9. 2025DRIVE: Dynamic Rule Inference and Verified Evaluation for Constraint-Aware Autonomous Drivingcited by 0
  10. 2026Diffusion-Based Data-Driven Assortment Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019Generalized Maximum Causal Entropy for Inverse Reinforcement Learning
  2. 2026Fast Rates for Inverse Reinforcement Learning
  3. 2022Weighted Maximum Entropy Inverse Reinforcement Learning
  4. 2024Convergence of a model-free entropy-regularized inverse reinforcement learning algorithm
  5. 2025Statistical analysis of Inverse Entropy-regularized Reinforcement Learning
  6. 2019Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning
  7. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations
  8. 2021Inverse Reinforcement Learning with Explicit Policy Estimates