# A Primer on Maximum Causal Entropy Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/2203.11409 · HTML: https://mlsanity.com/paper/2203.11409/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 17 of 17)

- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning](https://mlsanity.com/paper/1809.02925/) (2018)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Taming the Noise in Reinforcement Learning via Soft Updates](https://mlsanity.com/paper/1512.08562/) (2015)
- [Reward-rational (implicit) choice: A unifying formalism for reward learning](https://mlsanity.com/paper/2002.04833/) (2020)
- [Quantifying Differences in Reward Functions](https://mlsanity.com/paper/2006.13900/) (2020)
- [Identifiability in inverse reinforcement learning](https://mlsanity.com/paper/2106.03498/) (2021)
- [Invariance in Policy Optimisation and Partial Identifiability in Reward Learning](https://mlsanity.com/paper/2203.07475/) (2022)
- [A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learning](https://mlsanity.com/paper/1810.01531/) (2018)
- [Preferences Implicit in the State of the World](https://mlsanity.com/paper/1902.04198/) (2019)
- [Repeated Inverse Reinforcement Learning](https://mlsanity.com/paper/1705.05427/) (2017)
- [Inverse reinforcement learning for video games](https://mlsanity.com/paper/1810.10593/) (2018)

## Built on by (most cited first, 13 of 13)

- [On the Sensitivity of Reward Inference to Misspecified Human Models](https://mlsanity.com/paper/2212.04717/) (2022)
- [A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challenges](https://mlsanity.com/paper/2409.07569/) (2024)
- [A Bayesian Approach to Robust Inverse Reinforcement Learning](https://mlsanity.com/paper/2309.08571/) (2023)
- [Maximum Causal Entropy Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2305.02857/) (2023)
- [Measuring Goal-Directedness](https://mlsanity.com/paper/2412.04758/) (2024)
- [Maximum Causal Entropy IRL in Mean-Field Games and GNEP Framework for Forward RL](https://mlsanity.com/paper/2401.06566/) (2024)
- [Kernel Based Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games](https://mlsanity.com/paper/2507.14529/) (2025)
- [Probabilistic inverse optimal control for non-linear partially observable systems disentangles perceptual uncertainty and behavioral costs](https://mlsanity.com/paper/2303.16698/) (2023)
- [Efficient Reward Identification In Max Entropy Reinforcement Learning with Sparsity and Rank Priors](https://mlsanity.com/paper/2508.07400/) (2025)
- [Learning from Preferences and Mixed Demonstrations in General Settings](https://mlsanity.com/paper/2508.14027/) (2025)
- [Distributional Inverse Reinforcement Learning](https://mlsanity.com/paper/2510.03013/) (2025)
- [Incoherence in Goal-Conditioned Autoregressive Models](https://mlsanity.com/paper/2510.06545/) (2025)
- [Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward](https://mlsanity.com/paper/2606.16759/) (2026)

## Abstract (arXiv)

Inverse Reinforcement Learning (IRL) algorithms infer a reward function that explains demonstrations provided by an expert acting in the environment. Maximum Causal Entropy (MCE) IRL is currently the most popular formulation of IRL, with numerous extensions. In this tutorial, we present a compressed derivation of MCE IRL and the key results from contemporary implementations of MCE IRL algorithms. We hope this will serve both as an introductory resource for those new to the field, and as a concise reference for those already familiar with these topics.