# Multi-Agent Adversarial Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/1907.13220 · HTML: https://mlsanity.com/paper/1907.13220/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 20 of its 60 references that are in the map.
- First posted on arXiv 30 Jul 2019.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 20 of 20)

- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Inverse Reward Design](https://mlsanity.com/paper/1711.02827/) (2017)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation](https://mlsanity.com/paper/1708.05144/) (2017)
- [Multi-Agent Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1807.09936/) (2018)
- [Coordinated Multi-Agent Imitation Learning](https://mlsanity.com/paper/1703.03121/) (2017)
- [Model-Free Imitation Learning with Policy Optimization](https://mlsanity.com/paper/1605.08478/) (2016)
- [Computational Rationalization: The Inverse Equilibrium Problem](https://mlsanity.com/paper/1308.3506/) (2013)
- [Inverse Reinforcement Learning in Swarm Systems](https://mlsanity.com/paper/1602.05450/) (2016)
- [Multi-agent Inverse Reinforcement Learning for Certain General-sum Stochastic Games](https://mlsanity.com/paper/1806.09795/) (2018)

## Built on by (most cited first, 30 of 60)

- [Modeling Human Driving Behavior through Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2006.06412/) (2020)
- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions](https://mlsanity.com/paper/2402.13777/) (2024)
- [A General Recipe for Likelihood-free Bayesian Optimization](https://mlsanity.com/paper/2206.13035/) (2022)
- [OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching](https://mlsanity.com/paper/2109.04307/) (2021)
- [Multi-Agent Imitation Learning with Copulas](https://mlsanity.com/paper/2107.04750/) (2021)
- [Conditional Imitation Learning for Multi-Agent Games](https://mlsanity.com/paper/2201.01448/) (2022)
- [Multiagent Inverse Reinforcement Learning via Theory of Mind Reasoning](https://mlsanity.com/paper/2302.10238/) (2023)
- [Individual-Level Inverse Reinforcement Learning for Mean Field Games](https://mlsanity.com/paper/2202.06401/) (2022)
- [Adversarial Inverse Reinforcement Learning for Mean Field Games](https://mlsanity.com/paper/2104.14654/) (2021)
- [Learning Altruistic Behaviours in Reinforcement Learning without External Rewards](https://mlsanity.com/paper/2107.09598/) (2021)
- [On-the-fly Strategy Adaptation for ad-hoc Agent Coordination](https://mlsanity.com/paper/2203.08015/) (2022)
- [Scalable Multi-Agent Inverse Reinforcement Learning via Actor-Attention-Critic](https://mlsanity.com/paper/2002.10525/) (2020)
- [Grounded Relational Inference: Domain Knowledge Driven Explainable Autonomous Driving](https://mlsanity.com/paper/2102.11905/) (2021)
- [Learning Human Rewards by Inferring Their Latent Intelligence Levels in Multi-Agent Games: A Theory-of-Mind Approach with Application to Driving Data](https://mlsanity.com/paper/2103.04289/) (2021)
- [Multi-Agent Inverse Reinforcement Learning in Real World Unstructured Pedestrian Crowds](https://mlsanity.com/paper/2405.16439/) (2024)
- [Grounded Answers for Multi-agent Decision-making Problem through Generative World Model](https://mlsanity.com/paper/2410.02664/) (2024)
- [Synthetic Data-Based Simulators for Recommender Systems: A Survey](https://mlsanity.com/paper/2206.11338/) (2022)
- [Curricular Subgoals for Inverse Reinforcement Learning](https://mlsanity.com/paper/2306.08232/) (2023)
- [DecompGAIL: Learning Realistic Traffic Behaviors with Decomposed Multi-Agent Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2510.06913/) (2025)
- [Multi-Agent Advisor Q-Learning](https://mlsanity.com/paper/2111.00345/) (2021)
- [Robust Entropy-regularized Markov Decision Processes](https://mlsanity.com/paper/2112.15364/) (2021)
- [Select to Perfect: Imitating desired behavior from large multi-agent data](https://mlsanity.com/paper/2405.03735/) (2024)
- [On Feasible Rewards in Multi-Agent Inverse Reinforcement Learning](https://mlsanity.com/paper/2411.15046/) (2024)
- [Learning Equilibria from Data: Provably Efficient Multi-Agent Imitation Learning](https://mlsanity.com/paper/2505.17610/) (2025)
- [Equilibrium Inverse Reinforcement Learning for Ride-hailing Vehicle Network](https://mlsanity.com/paper/2102.06854/) (2021)
- [The Multi-Agent Pickup and Delivery Problem: MAPF, MARL and Its Warehouse Applications](https://mlsanity.com/paper/2203.07092/) (2022)
- [Identifying Coordination in a Cognitive Radar Network -- A Multi-Objective Inverse Reinforcement Learning Approach](https://mlsanity.com/paper/2211.06967/) (2022)
- [Utility Theory based Cognitive Modeling in the Application of Robotics: A Survey](https://mlsanity.com/paper/2306.09445/) (2023)
- [Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review](https://mlsanity.com/paper/2411.10268/) (2024)

## Abstract (arXiv)

Reinforcement learning agents are prone to undesired behaviors due to reward mis-specification. Finding a set of reward functions to properly guide agent behaviors is particularly challenging in multi-agent scenarios. Inverse reinforcement learning provides a framework to automatically acquire suitable reward functions from expert demonstrations. Its extension to multi-agent settings, however, is difficult due to the more complex notions of rational behaviors. In this paper, we propose MA-AIRL, a new framework for multi-agent inverse reinforcement learning, which is effective and scalable for Markov games with high-dimensional state-action space and unknown dynamics. We derive our algorithm based on a new solution concept and maximum pseudolikelihood estimation within an adversarial reward learning framework. In the experiments, we demonstrate that MA-AIRL can recover reward functions that are highly correlated with ground truth ones, and significantly outperforms prior methods in terms of policy imitation.