# Data-Driven Inverse Reinforcement Learning for Expert-Learner Zero-Sum Games

arXiv: https://arxiv.org/abs/2301.01997 · HTML: https://mlsanity.com/paper/2301.01997/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 4 of its 42 references that are in the map.
- First posted on arXiv 5 Jan 2023.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 4 of 4)

- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Online inverse reinforcement learning with unknown disturbances](https://mlsanity.com/paper/2003.03912/) (2020)
- [Online Inverse Optimal Control for Control-Constrained Discrete-Time Systems on Finite and Infinite Horizons](https://mlsanity.com/paper/2005.06153/) (2020)
- [Online Observer-Based Inverse Reinforcement Learning](https://mlsanity.com/paper/2011.02057/) (2020)

## Built on by (most cited first, 2 of 2)

- [Inverse reinforcement learning by expert imitation for the stochastic linear-quadratic optimal control problem](https://mlsanity.com/paper/2405.17085/) (2024)
- [Inverse Optimal Control for Linear Quadratic Problem with Poisson Jumps: Model-Free Inverse Reinforcement Learning Approaches](https://mlsanity.com/paper/2607.03169/) (2026)

## Abstract (arXiv)

In this paper, we formulate inverse reinforcement learning (IRL) as an expert-learner interaction whereby the optimal performance intent of an expert or target agent is unknown to a learner agent. The learner observes the states and controls of the expert and hence seeks to reconstruct the expert's cost function intent and thus mimics the expert's optimal response. Next, we add non-cooperative disturbances that seek to disrupt the learning and stability of the learner agent. This leads to the formulation of a new interaction we call zero-sum game IRL. We develop a framework to solve the zero-sum game IRL problem that is a modified extension of RL policy iteration (PI) to allow unknown expert performance intentions to be computed and non-cooperative disturbances to be rejected. The framework has two parts: a value function and control action update based on an extension of PI, and a cost function update based on standard inverse optimal control. Then, we eventually develop an off-policy IRL algorithm that does not require knowledge of the expert and learner agent dynamics and performs single-loop learning. Rigorous proofs and analyses are given. Finally, simulation experiments are presented to show the effectiveness of the new approach.