# TiZero: Mastering Multi-Agent Football with Curriculum Learning and Self-Play

arXiv: https://arxiv.org/abs/2302.07515 · HTML: https://mlsanity.com/paper/2302.07515/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 78 references that are in the map.
- First posted on arXiv 15 Feb 2023.

## Builds on (most cited first, 30 of 42)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.05408/) (2019)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [QPLEX: Duplex Dueling Multi-Agent Q-Learning](https://mlsanity.com/paper/2008.01062/) (2020)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Google Research Football: A Novel Reinforcement Learning Environment](https://mlsanity.com/paper/1907.11180/) (2019)
- [The Complexity of Decentralized Control of Markov Decision Processes](https://mlsanity.com/paper/1301.3836/) (2013)
- [ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning](https://mlsanity.com/paper/1605.02097/) (2016)
- [Multi-Agent Reinforcement Learning is a Sequence Modeling Problem](https://mlsanity.com/paper/2205.14953/) (2022)
- [RODE: Learning Roles to Decompose Multi-Agent Tasks](https://mlsanity.com/paper/2010.01523/) (2020)
- [Effective Diversity in Population Based Reinforcement Learning](https://mlsanity.com/paper/2002.00632/) (2020)
- [Celebrating Diversity in Shared Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.02195/) (2021)
- [Open-ended Learning in Symmetric Zero-sum Games](https://mlsanity.com/paper/1901.08106/) (2019)
- [From Motor Control to Team Play in Simulated Humanoid Football](https://mlsanity.com/paper/2105.12196/) (2021)
- [Towards Playing Full MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/2011.12692/) (2020)
- [Real World Games Look Like Spinning Tops](https://mlsanity.com/paper/2004.09468/) (2020)
- [DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning](https://mlsanity.com/paper/2106.06135/) (2021)
- [Emergent Coordination Through Competition](https://mlsanity.com/paper/1902.07151/) (2019)

## Built on by (most cited first, 20 of 20)

- [A Survey on Self-play Methods in Reinforcement Learning](https://mlsanity.com/paper/2408.01072/) (2024)
- [N-Agent Ad Hoc Teamwork](https://mlsanity.com/paper/2404.10740/) (2024)
- [A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decision](https://mlsanity.com/paper/2307.03353/) (2023)
- [Reinforcement Learning Within the Classical Robotics Stack: A Case Study in Robot Soccer](https://mlsanity.com/paper/2412.09417/) (2024)
- [ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamwork](https://mlsanity.com/paper/2505.23686/) (2025)
- [Multi-Agent Diagnostics for Robustness via Illuminated Diversity](https://mlsanity.com/paper/2401.13460/) (2024)
- [The impact of behavioral diversity in multi-agent reinforcement learning](https://mlsanity.com/paper/2412.16244/) (2024)
- [OpenRL: A Unified Reinforcement Learning Framework](https://mlsanity.com/paper/2312.16189/) (2023)
- [SPECTra: Scalable Multi-Agent Reinforcement Learning with Permutation-Free Networks](https://mlsanity.com/paper/2503.11726/) (2025)
- [CRAFT: Coaching Reinforcement Learning Autonomously using Foundation Models for Multi-Robot Coordination Tasks](https://mlsanity.com/paper/2509.14380/) (2025)
- [Improving Sample Efficiency in Multi-Agent Reinforcement Learning for Simulated Football Games via Exploration](https://mlsanity.com/paper/2503.13077/) (2025)
- [Expandable Decision-Making States for Multi-Agent Deep Reinforcement Learning in Soccer Tactical Analysis](https://mlsanity.com/paper/2510.00480/) (2025)
- [DCIR: Dynamic Consistency Intrinsic Reward for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2312.05783/) (2023)
- [Transformer Guided Coevolution: Improved Team Selection in Multiagent Adversarial Team Games](https://mlsanity.com/paper/2410.13769/) (2024)
- [Generalizable Agent Modeling for Agent Collaboration-Competition Adaptation with Multi-Retrieval and Dynamic Generation](https://mlsanity.com/paper/2506.16718/) (2025)
- [Understanding Action Effects through Instrumental Empowerment in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2508.15652/) (2025)
- [ReCollab: Retrieval-Augmented LLMs for Cooperative Ad-hoc Teammate Modeling](https://mlsanity.com/paper/2512.22129/) (2025)
- [NePPO: Near-Potential Policy Optimization for General-Sum Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2603.06977/) (2026)
- [Reinforcement Learning Enabled Adaptive Multi-Task Control for Bipedal Soccer Robots](https://mlsanity.com/paper/2604.19104/) (2026)
- [Strategic Multi-Agent Learning for Interpretable Action Valuation of All Players in Football](https://mlsanity.com/paper/2610.05961/) (2026)

## Abstract (arXiv)

Multi-agent football poses an unsolved challenge in AI research. Existing work has focused on tackling simplified scenarios of the game, or else leveraging expert demonstrations. In this paper, we develop a multi-agent system to play the full 11 vs. 11 game mode, without demonstrations. This game mode contains aspects that present major challenges to modern reinforcement learning algorithms; multi-agent coordination, long-term planning, and non-transitivity. To address these challenges, we present TiZero; a self-evolving, multi-agent system that learns from scratch. TiZero introduces several innovations, including adaptive curriculum learning, a novel self-play strategy, and an objective that optimizes the policies of multiple agents jointly. Experimentally, it outperforms previous systems by a large margin on the Google Research Football environment, increasing win rates by over 30%. To demonstrate the generality of TiZero's innovations, they are assessed on several environments beyond football; Overcooked, Multi-agent Particle-Environment, Tic-Tac-Toe and Connect-Four.