# Soft-Bellman Equilibrium in Affine Markov Games: Forward Solutions and Inverse Learning

arXiv: https://arxiv.org/abs/2304.00163 · HTML: https://mlsanity.com/paper/2304.00163/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 31 Mar 2023.

## Builds on (most cited first, 9 of 9)

- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [Coordinated Multi-Agent Imitation Learning](https://mlsanity.com/paper/1703.03121/) (2017)
- [JuMP: A Modeling Language for Mathematical Optimization](https://mlsanity.com/paper/1508.01982/) (2015)
- [Computational Rationalization: The Inverse Equilibrium Problem](https://mlsanity.com/paper/1308.3506/) (2013)
- [Maximum-Entropy Multi-Agent Dynamic Games: Forward and Inverse Solutions](https://mlsanity.com/paper/2110.01027/) (2021)
- [Inverse Reinforcement Learning in Swarm Systems](https://mlsanity.com/paper/1602.05450/) (2016)
- [Multi-agent Inverse Reinforcement Learning for Two-person Zero-sum Games](https://mlsanity.com/paper/1403.6508/) (2014)
- [Inverse Matrix Games with Unique Quantal Response Equilibrium](https://mlsanity.com/paper/2207.08275/) (2022)

## Built on by (most cited first, 1 of 1)

- [Active Inverse Learning in Stackelberg Trajectory Games](https://mlsanity.com/paper/2308.08017/) (2023)

## Abstract (arXiv)

Markov games model interactions among multiple players in a stochastic, dynamic environment. Each player in a Markov game maximizes its expected total discounted reward, which depends upon the policies of the other players. We formulate a class of Markov games, termed affine Markov games, where an affine reward function couples the players' actions. We introduce a novel solution concept, the soft-Bellman equilibrium, where each player is boundedly rational and chooses a soft-Bellman policy rather than a purely rational policy as in the well-known Nash equilibrium concept. We provide conditions for the existence and uniqueness of the soft-Bellman equilibrium and propose a nonlinear least-squares algorithm to compute such an equilibrium in the forward problem. We then solve the inverse game problem of inferring the players' reward parameters from observed state-action trajectories via a projected-gradient algorithm. Experiments in a predator-prey OpenAI Gym environment show that the reward parameters inferred by the proposed algorithm outperform those inferred by a baseline algorithm: they reduce the Kullback-Leibler divergence between the equilibrium policies and observed policies by at least two orders of magnitude.