# Kernel Based Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games

arXiv: https://arxiv.org/abs/2507.14529 · HTML: https://mlsanity.com/paper/2507.14529/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 10 of its 31 references that are in the map.
- First posted on arXiv 19 Jul 2025.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 10 of 10)

- [A unified view of entropy-regularized Markov decision processes](https://mlsanity.com/paper/1705.07798/) (2017)
- [Learning in Mean Field Games: A Survey](https://mlsanity.com/paper/2205.12944/) (2022)
- [A Primer on Maximum Causal Entropy Inverse Reinforcement Learning](https://mlsanity.com/paper/2203.11409/) (2022)
- [Individual-Level Inverse Reinforcement Learning for Mean Field Games](https://mlsanity.com/paper/2202.06401/) (2022)
- [Learning Deep Mean Field Games for Modeling Large Population Behavior](https://mlsanity.com/paper/1711.03156/) (2017)
- [Adversarial Inverse Reinforcement Learning for Mean Field Games](https://mlsanity.com/paper/2104.14654/) (2021)
- [Multi-Agent Inverse Reinforcement Learning in Real World Unstructured Pedestrian Crowds](https://mlsanity.com/paper/2405.16439/) (2024)
- [Kernel-Based Function Approximation for Average Reward Reinforcement Learning: An Optimist No-Regret Algorithm](https://mlsanity.com/paper/2410.23498/) (2024)
- [Linearly Solvable Mean-Field Traffic Routing Games](https://mlsanity.com/paper/1903.01449/) (2019)
- [Maximum Causal Entropy IRL in Mean-Field Games and GNEP Framework for Forward RL](https://mlsanity.com/paper/2401.06566/) (2024)

## Built on by (most cited first, 3 of 3)

- [Maximum Entropy Inverse Reinforcement Learning for Mean-Field Games with Average Reward](https://mlsanity.com/paper/2606.16759/) (2026)
- [Neural Parameter Calibration for Finite-State Mean Field Games](https://mlsanity.com/paper/2606.23155/) (2026)
- [Inverse Learning of the Altruism and Cost Level in Mixed-Individual Mean Field Games](https://mlsanity.com/paper/2609.13469/) (2026)

## Abstract (arXiv)

We consider the maximum causal entropy inverse reinforcement learning (IRL) problem for infinite-horizon stationary mean-field games (MFG), in which we model the unknown reward function within a reproducing kernel Hilbert space (RKHS). This allows the inference of rich and potentially nonlinear reward structures directly from expert demonstrations, in contrast to most existing approaches for MFGs that typically restrict the reward to a linear combination of a fixed finite set of basis functions and rely on finite-horizon formulations. We introduce a Lagrangian relaxation that enables us to reformulate the problem as an unconstrained log-likelihood maximization and obtain a solution via a gradient ascent algorithm. To establish the theoretical consistency of the algorithm, we prove the smoothness of the log-likelihood objective through the Fr'echet differentiability of the related soft Bellman operators with respect to the parameters in the RKHS. To illustrate the practical advantages of the RKHS formulation, we validate our framework on a mean-field traffic routing game exhibiting state-dependent preference reversal, where the kernel-based method reduces policy recovery error by over an order of magnitude compared to a linear reward baseline with a comparable parameter count. Furthermore, we extend the framework to the finite-horizon non-stationary setting. We demonstrate that the log-likelihood reformulation is structurally unavailable in this regime and instead develop an alternative gradient descent algorithm on the convex dual via Danskin's theorem, establishing smoothness and convergence guarantees.