# Maximum Causal Entropy IRL in Mean-Field Games and GNEP Framework for Forward RL

arXiv: https://arxiv.org/abs/2401.06566 · HTML: https://mlsanity.com/paper/2401.06566/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 13 of its 36 references that are in the map.
- First posted on arXiv 12 Jan 2024.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 13 of 13)

- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [A unified view of entropy-regularized Markov decision processes](https://mlsanity.com/paper/1705.07798/) (2017)
- [Handbook of Convergence Theorems for (Stochastic) Gradient Methods](https://mlsanity.com/paper/2301.11235/) (2023)
- [Approximately Solving Mean Field Games via Entropy-Regularized Deep Reinforcement Learning](https://mlsanity.com/paper/2102.01585/) (2021)
- [Learning in Mean Field Games: A Survey](https://mlsanity.com/paper/2205.12944/) (2022)
- [Q-Learning in Regularized Mean-field Games](https://mlsanity.com/paper/2003.12151/) (2020)
- [A Primer on Maximum Causal Entropy Inverse Reinforcement Learning](https://mlsanity.com/paper/2203.11409/) (2022)
- [Revisiting Maximum Entropy Inverse Reinforcement Learning: New Perspectives and Algorithms](https://mlsanity.com/paper/2012.00889/) (2020)
- [Individual-Level Inverse Reinforcement Learning for Mean Field Games](https://mlsanity.com/paper/2202.06401/) (2022)
- [On Imitation in Mean-field Games](https://mlsanity.com/paper/2306.14799/) (2023)
- [Learning Deep Mean Field Games for Modeling Large Population Behavior](https://mlsanity.com/paper/1711.03156/) (2017)
- [Adversarial Inverse Reinforcement Learning for Mean Field Games](https://mlsanity.com/paper/2104.14654/) (2021)
- [Linear Mean-Field Games with Discounted Cost](https://mlsanity.com/paper/2301.06074/) (2023)

## Built on by (most cited first, 4 of 4)

- [Kernel Based Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games](https://mlsanity.com/paper/2507.14529/) (2025)
- [Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward](https://mlsanity.com/paper/2606.16759/) (2026)
- [Neural Parameter Calibration for Finite-State Mean Field Games](https://mlsanity.com/paper/2606.23155/) (2026)
- [Inverse Learning of the Altruism and Cost Level in Mixed-Individual Mean Field Games](https://mlsanity.com/paper/2609.13469/) (2026)

## Abstract (arXiv)

This paper explores the use of Maximum Causal Entropy Inverse Reinforcement Learning (IRL) within the context of discrete-time stationary Mean-Field Games (MFGs) characterized by finite state spaces and an infinite-horizon, discounted-reward setting. Although the resulting optimization problem is non-convex with respect to policies, we reformulate it as a convex optimization problem in terms of state-action occupation measures by leveraging the linear programming framework of Markov Decision Processes. Based on this convex reformulation, we introduce a gradient descent algorithm with a guaranteed convergence rate to efficiently compute the optimal solution. Moreover, we develop a new method that conceptualizes the MFG problem as a Generalized Nash Equilibrium Problem (GNEP), enabling effective computation of the mean-field equilibrium for forward reinforcement learning (RL) problems and marking an advancement in MFG solution techniques. We further illustrate the practical applicability of our GNEP approach by employing this algorithm to generate data for numerical MFG examples.