mlsanityData to 7 Oct 2026

Open-ended Learning in Symmetric Zero-sum Games

arXiv 1901.08106 ↗First posted 23 Jan 2019cs.LG cs.GT cs.MAcited by 96

Not on the map. Why?

Abstract (arXiv)

Zero-sum games such as chess and poker are, abstractly, functions that evaluate pairs of agents, for example labeling them `winner' and `loser'. If the game is approximately transitive, then self-play generates sequences of agents of increasing strength. However, nontransitive games, such as rock-paper-scissors, can exhibit strategic cycles, and there is no longer a clear objective -- we want agents to increase in strength, but against whom is unclear. In this paper, we introduce a geometric framework for formulating agent objectives in zero-sum games, in order to construct adaptive sequences of objectives that yield open-ended learning. The framework allows us to reason about population performance in nontransitive games, and enables the development of a new algorithm (rectified Nash response, PSRO_rN) that uses game-theoretic niching to construct diverse populations of effective agents, producing a stronger set of agents than existing algorithms. We apply PSRO_rN to two highly nontransitive resource allocation games and find that PSRO_rN consistently outperforms the existing alternatives.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 0; 2021: 23; 2022: 18; 2023: 19; 2024: 13; 2025: 11; 2026: 11.

96 papers.

Sort:

By area:

  1. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  2. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 111
  3. 2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 97
  4. 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
  5. 2021Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordinationcited by 69
  6. 2021Modelling Behavioural Diversity for Learning in Open-Ended Gamescited by 39
  7. 2022Melting Pot 2.0cited by 37
  8. 2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Playcited by 37
  9. 2021Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomizationcited by 35
  10. 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 33
  11. 2022Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learningcited by 32
  12. 2023Cooperative Open-ended Learning Framework for Zero-shot Coordinationcited by 30
38 more
  1. 2023MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learningcited by 30
  2. 2022Adversarial Policies Beat Superhuman Go AIscited by 29
  3. 2023A Review of Cooperation in Multi-agent Learningcited by 27
  4. 2021Which priors matter? Benchmarking models for learning latent dynamicscited by 26
  5. 2024A Survey on Self-play Methods in Reinforcement Learningcited by 26
  6. 2025Large Language Models Miss the Multi-Agent Markcited by 25
  7. 2021Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systemscited by 24
  8. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 24
  9. 2022On the Limitations of Elo: Real-World Games, are Transitive, not Additivecited by 23
  10. 2021Pick Your Battles: Interaction Graphs as Population-Level Objectives for Strategic Diversitycited by 22
  11. 2021Quantifying the effects of environment and population diversity in multi-agent reinforcement learningcited by 21
  12. 2021Multi-Agent Training beyond Zero-Sum with Correlated Equilibrium Meta-Solverscited by 21
  13. 2024A social path to human-like artificial intelligencecited by 20
  14. 2021Online Double Oraclecited by 18
  15. 2023TiZero: Mastering Multi-Agent Football with Curriculum Learning and Self-Playcited by 18
  16. 2023Policy Space Diversity for Non-Transitive Gamescited by 15
  17. 2021Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Gamescited by 14
  18. 2021A Game-Theoretic Approach for Improving Generalization Ability of TSP Solverscited by 14
  19. 2022NeuPL: Neural Population Learningcited by 14
  20. 2024Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learningcited by 14
  21. 2023Tackling Cooperative Incompatibility for Zero-Shot Human-AI Coordinationcited by 12
  22. 2021Complex Momentum for Optimization in Gamescited by 11
  23. 2022Learning Representations that Enable Generalization in Assistive Taskscited by 11
  24. 2021Measuring the Non-Transitivity in Chesscited by 10
  25. 2022Towards Multi-Agent Reinforcement Learning driven Over-The-Counter Market Simulationscited by 10
  26. 2024Policy Space Response Oracles: A Surveycited by 10
  27. 2022Anytime PSRO for Two-Player Zero-Sum Gamescited by 9
  28. 2021Lyapunov Exponents for Diversity in Differentiable Gamescited by 8
  29. 2022Efficient Policy Space Response Oraclescited by 8
  30. 2022Simplex Neural Population Learning: Any-Mixture Bayes-Optimality in Symmetric Zero-sum Gamescited by 8
  31. 2021A Game-Theoretic Approach to Multi-Agent Trust Region Optimizationcited by 7
  32. 2021Learning Diverse Policies in MOBA Games via Macro-Goalscited by 7
  33. 2023Rethinking Adversarial Policies: A Generalized Attack Formulation and Provable Defense in RLcited by 7
  34. 2024Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planningcited by 7
  35. 2023Learning Diverse Risk Preferences in Population-based Self-playcited by 6
  36. 2023Fictitious Cross-Play: Learning Global Nash Equilibrium in Mixed Cooperative-Competitive Gamescited by 6
  37. 2023Building Open-Ended Embodied Agent via Language-Policy Bidirectional Adaptationcited by 6
  38. 2023Robust Driving Policy Learning with Guided Meta Reinforcement Learningcited by 5

50 most cited of 96 All 96 (JSON)