# Empirical Policy Optimization for n-Player Markov Games

arXiv: https://arxiv.org/abs/2110.08979 · HTML: https://mlsanity.com/paper/2110.08979/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Oct 2021.

## Built on by (most cited first, 4 of 4)

- [A Hierarchical Deep Reinforcement Learning Framework for 6-DOF UCAV Air-to-Air Combat](https://mlsanity.com/paper/2212.03830/) (2022)
- [Self-evolving Autoencoder Embedded Q-Network](https://mlsanity.com/paper/2402.11604/) (2024)
- [ComSD: Balancing Behavioral Quality and Diversity in Unsupervised Skill Discovery](https://mlsanity.com/paper/2309.17203/) (2023)
- [FM3Q: Factorized Multi-Agent MiniMax Q-Learning for Two-Team Zero-Sum Markov Game](https://mlsanity.com/paper/2402.00738/) (2024)

## Abstract (arXiv)

In single-agent Markov decision processes, an agent can optimize its policy based on the interaction with environment. In multi-player Markov games (MGs), however, the interaction is non-stationary due to the behaviors of other players, so the agent has no fixed optimization objective. In this paper, we treat the evolution of player policies as a dynamical process and propose a novel learning scheme for Nash equilibrium. The core is to evolve one's policy according to not just its current in-game performance, but an aggregation of its performance over history. We show that for a variety of MGs, players in our learning scheme will provably converge to a point that is an approximation to Nash equilibrium. Combined with neural networks, we develop the empirical policy optimization algorithm, that is implemented in a reinforcement-learning framework and runs in a distributed way, with each player optimizing its policy based on own observations. We use two numerical examples to validate the convergence property on small-scale MGs with n≥ 2 players, and a pong example to show the potential of our algorithm on large games.