# Competitive Multi-agent Inverse Reinforcement Learning with Sub-optimal Demonstrations

arXiv: https://arxiv.org/abs/1801.02124 · HTML: https://mlsanity.com/paper/1801.02124/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 7 of its 21 references that are in the map.
- First posted on arXiv 7 Jan 2018.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 7 of 7)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Learning with Opponent-Learning Awareness](https://mlsanity.com/paper/1709.04326/) (2017)
- [Density Matching Reward Learning](https://mlsanity.com/paper/1608.03694/) (2016)
- [A Study of Gradient Descent Schemes for General-Sum Stochastic Games](https://mlsanity.com/paper/1507.00093/) (2015)

## Built on by (most cited first, 11 of 11)

- [Multiagent Inverse Reinforcement Learning via Theory of Mind Reasoning](https://mlsanity.com/paper/2302.10238/) (2023)
- [Non-Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1911.04220/) (2019)
- [Multi-agent Inverse Reinforcement Learning for Certain General-sum Stochastic Games](https://mlsanity.com/paper/1806.09795/) (2018)
- [Online Observer-Based Inverse Reinforcement Learning](https://mlsanity.com/paper/2011.02057/) (2020)
- [Multi-Agent Advisor Q-Learning](https://mlsanity.com/paper/2111.00345/) (2021)
- [Multi-Agent Inverse Reinforcement Learning: Suboptimal Demonstrations and Alternative Solution Concepts](https://mlsanity.com/paper/2109.01178/) (2021)
- [Semi-Supervised Imitation Learning of Team Policies from Suboptimal Demonstrations](https://mlsanity.com/paper/2205.02959/) (2022)
- [Approximate State Abstraction for Markov Games](https://mlsanity.com/paper/2412.15877/) (2024)
- [Concurrent Learning with Aggregated States via Randomized Least Squares Value Iteration](https://mlsanity.com/paper/2501.13394/) (2025)
- [Blind Inverse Game Theory: Jointly Decoding Rewards and Rationality in Entropy-Regularized Competitive Games](https://mlsanity.com/paper/2511.05640/) (2025)
- [Interactive Inverse Reinforcement Learning of Interaction Scenarios via Bi-level Optimization](https://mlsanity.com/paper/2605.08131/) (2026)

## Abstract (arXiv)

This paper considers the problem of inverse reinforcement learning in zero-sum stochastic games when expert demonstrations are known to be not optimal. Compared to previous works that decouple agents in the game by assuming optimality in expert strategies, we introduce a new objective function that directly pits experts against Nash Equilibrium strategies, and we design an algorithm to solve for the reward function in the context of inverse reinforcement learning with deep neural networks as model approximations. In our setting the model and algorithm do not decouple by agent. In order to find Nash Equilibrium in large-scale games, we also propose an adversarial training algorithm for zero-sum stochastic games, and show the theoretical appeal of non-existence of local optima in its objective function. In our numerical experiments, we demonstrate that our Nash Equilibrium and inverse reinforcement learning algorithms address games that are not amenable to previous approaches using tabular representations. Moreover, with sub-optimal expert demonstrations our algorithms recover both reward functions and strategies with good quality.