# Multi-Agent Training for Pommerman: Curriculum Learning and Population-based Self-Play Approach

arXiv: https://arxiv.org/abs/2407.00662 · HTML: https://mlsanity.com/paper/2407.00662/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 7 references that are in the map.
- First posted on arXiv 30 Jun 2024.

## Builds on (most cited first, 5 of 5)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Emergent Complexity via Multi-Agent Competition](https://mlsanity.com/paper/1710.03748/) (2017)
- [Pommerman: A Multi-Agent Playground](https://mlsanity.com/paper/1809.07124/) (2018)
- [Skynet: A Top Deep RL Agent in the Inaugural Pommerman Team Competition](https://mlsanity.com/paper/1905.01360/) (2019)
- [On Hard Exploration for Reinforcement Learning: a Case Study in Pommerman](https://mlsanity.com/paper/1907.11788/) (2019)

## Built on by (most cited first, 3 of 3)

- [Among Us: A Sandbox for Measuring and Detecting Agentic Deception](https://mlsanity.com/paper/2504.04072/) (2025)
- [Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordia](https://mlsanity.com/paper/2512.03318/) (2025)
- [Curriculum-Based Iterative Self-Play for Scalable Multi-Drone Racing](https://mlsanity.com/paper/2510.22570/) (2025)

## Abstract (arXiv)

Pommerman is a multi-agent environment that has received considerable attention from researchers in recent years. This environment is an ideal benchmark for multi-agent training, providing a battleground for two teams with communication capabilities among allied agents. Pommerman presents significant challenges for model-free reinforcement learning due to delayed action effects, sparse rewards, and false positives, where opponent players can lose due to their own mistakes. This study introduces a system designed to train multi-agent systems to play Pommerman using a combination of curriculum learning and population-based self-play. We also tackle two challenging problems when deploying the multi-agent training system for competitive games: sparse reward and suitable matchmaking mechanism. Specifically, we propose an adaptive annealing factor based on agents' performance to adjust the dense exploration reward during training dynamically. Additionally, we implement a matchmaking mechanism utilizing the Elo rating system to pair agents effectively. Our experimental results demonstrate that our trained agent can outperform top learning agents without requiring communication among allied agents.