# NeuPL: Neural Population Learning

arXiv: https://arxiv.org/abs/2202.07415 · HTML: https://mlsanity.com/paper/2202.07415/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 15 Feb 2022.

## Builds on (most cited first, 13 of 13)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Open-ended Learning in Symmetric Zero-sum Games](https://mlsanity.com/paper/1901.08106/) (2019)
- [From Motor Control to Team Play in Simulated Humanoid Football](https://mlsanity.com/paper/2105.12196/) (2021)
- [Real World Games Look Like Spinning Tops](https://mlsanity.com/paper/2004.09468/) (2020)
- [Emergent Coordination Through Competition](https://mlsanity.com/paper/1902.07151/) (2019)
- [A Generalized Training Approach for Multiagent Learning](https://mlsanity.com/paper/1909.12823/) (2019)
- [Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games](https://mlsanity.com/paper/2006.08555/) (2020)
- [Pick Your Battles: Interaction Graphs as Population-Level Objectives for Strategic Diversity](https://mlsanity.com/paper/2110.04041/) (2021)
- [Multi-Agent Training beyond Zero-Sum with Correlated Equilibrium Meta-Solvers](https://mlsanity.com/paper/2106.09435/) (2021)
- [Iterative Empirical Game Solving via Single Policy Best Response](https://mlsanity.com/paper/2106.01901/) (2021)
- [Learning to Play against Any Mixture of Opponents](https://mlsanity.com/paper/2009.14180/) (2020)

## Built on by (most cited first, 14 of 14)

- [A Survey on Self-play Methods in Reinforcement Learning](https://mlsanity.com/paper/2408.01072/) (2024)
- [TiZero: Mastering Multi-Agent Football with Curriculum Learning and Self-Play](https://mlsanity.com/paper/2302.07515/) (2023)
- [Policy Space Diversity for Non-Transitive Games](https://mlsanity.com/paper/2306.16884/) (2023)
- [Diversifying AI: Towards Creative Chess with AlphaZero](https://mlsanity.com/paper/2308.09175/) (2023)
- [Turbocharging Solution Concepts: Solving NEs, CEs and CCEs with Neural Equilibrium Solvers](https://mlsanity.com/paper/2210.09257/) (2022)
- [Simplex Neural Population Learning: Any-Mixture Bayes-Optimality in Symmetric Zero-sum Games](https://mlsanity.com/paper/2205.15879/) (2022)
- [Deep Reinforcement Learning for Autonomous Cyber Defence: A Survey](https://mlsanity.com/paper/2310.07745/) (2023)
- [Learning Diverse Risk Preferences in Population-based Self-play](https://mlsanity.com/paper/2305.11476/) (2023)
- [Feasible Adversarial Robust Reinforcement Learning for Underspecified Environments](https://mlsanity.com/paper/2207.09597/) (2022)
- [Developing, Evaluating and Scaling Learning Agents in Multi-Agent Environments](https://mlsanity.com/paper/2209.10958/) (2022)
- [Rethinking Teacher-Student Curriculum Learning through the Cooperative Mechanics of Experience](https://mlsanity.com/paper/2404.03084/) (2024)
- [DiffFP: Learning Behaviors from Scratch via Diffusion-based Fictitious Play](https://mlsanity.com/paper/2511.13186/) (2025)
- [Data-Augmented Game Starts for Accelerating Self-Play Exploration in Imperfect Information Games](https://mlsanity.com/paper/2605.14379/) (2026)
- [EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games](https://mlsanity.com/paper/2606.23995/) (2026)

## Abstract (arXiv)

Learning in strategy games (e.g. StarCraft, poker) requires the discovery of diverse policies. This is often achieved by iteratively training new policies against existing ones, growing a policy population that is robust to exploit. This iterative approach suffers from two issues in real-world games: a) under finite budget, approximate best-response operators at each iteration needs truncating, resulting in under-trained good-responses populating the population; b) repeated learning of basic skills at each iteration is wasteful and becomes intractable in the presence of increasingly strong opponents. In this work, we propose Neural Population Learning (NeuPL) as a solution to both issues. NeuPL offers convergence guarantees to a population of best-responses under mild assumptions. By representing a population of policies within a single conditional model, NeuPL enables transfer learning across policies. Empirically, we show the generality, improved performance and efficiency of NeuPL across several test domains. Most interestingly, we show that novel strategies become more accessible, not less, as the neural population expands.