mlsanityData to 7 Oct 2026

Self-Play PSRO: Toward Optimal Populations in Two-Player Zero-Sum Games

arXiv 2207.06541 ↗First posted 13 Jul 2022cs.GT cs.LG cs.MAcited by 11

Not on the map. Why?

Abstract (arXiv)

In competitive two-agent environments, deep reinforcement learning (RL) methods based on the Double Oracle (DO) algorithm, such as Policy Space Response Oracles (PSRO) and Anytime PSRO (APSRO), iteratively add RL best response policies to a population. Eventually, an optimal mixture of these population policies will approximate a Nash equilibrium. However, these methods might need to add all deterministic policies before converging. In this work, we introduce Self-Play PSRO (SP-PSRO), a method that adds an approximately optimal stochastic policy to the population in each iteration. Instead of adding only deterministic best responses to the opponent's least exploitable population mixture, SP-PSRO also learns an approximately optimal stochastic policy and adds it to the population as well. As a result, SP-PSRO empirically tends to converge much faster than APSRO and in many games converges in just a few iterations.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 28 of its 44 references that are in the map. First posted on arXiv 13 Jul 2022.

Builds on

28 of 44 references.

References · 28

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  3. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  4. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 180
  5. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 153
  6. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 129
  7. 2021V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RLcited by 111
  8. 2018Deep Counterfactual Regret Minimizationcited by 102
  9. 2020Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibriumcited by 100
  10. 2021Independent Policy Gradient Methods for Competitive Reinforcement Learningcited by 97
  11. 2021Global Convergence of Multi-Agent Policy Gradient in Markov Potential Gamescited by 89
  12. 2017Online Reinforcement Learning in Stochastic Gamescited by 67
16 more
  1. 2022A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Gamescited by 64
  2. 2020From Poincar'e Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularizationcited by 59
  3. 2022Independent Policy Gradient for Large-Scale Markov Potential Games: Sharper Rates, Function Approximation, and Game-Agnostic Convergencecited by 54
  4. 2019A Generalized Training Approach for Multiagent Learningcited by 53
  5. 2020Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Gamescited by 47
  6. 2021Modelling Behavioural Diversity for Learning in Open-Ended Gamescited by 39
  7. 2020DREAM: Deep Regret minimization with Advantage baselines and Model-free learningcited by 36
  8. 2018Actor-Critic Policy Optimization in Partially Observable Multiagent Environmentscited by 35
  9. 2021XDO: A Double Oracle Algorithm for Extensive-Form Gamescited by 34
  10. 2021Independent Natural Policy Gradient Always Converges in Markov Potential Gamescited by 34
  11. 2021Gradient play in stochastic games: stationary points, convergence, and sample complexitycited by 32
  12. 2021Learning in Nonzero-Sum Stochastic Games with Potentialscited by 25
  13. 2022ESCHER: Eschewing Importance Sampling in Games by Computing a History Value Function to Estimate Regretcited by 24
  14. 2021Multi-Agent Training beyond Zero-Sum with Correlated Equilibrium Meta-Solverscited by 21
  15. 2021Online Double Oraclecited by 18
  16. 2022Anytime PSRO for Two-Player Zero-Sum Gamescited by 9

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 6; 2024: 2; 2025: 2.

11 papers.

  1. 2025Language Self-Play For Data-Free Trainingcited by 42
  2. 2023Cooperative Open-ended Learning Framework for Zero-shot Coordinationcited by 30
  3. 2023ASP: Learn a Universal Neural Solver!cited by 27
  4. 2024A Survey on Self-play Methods in Reinforcement Learningcited by 26
  5. 2025Reevaluating Policy Gradient Methods for Imperfect-Information Gamescited by 13
  6. 2023Tackling Cooperative Incompatibility for Zero-Shot Human-AI Coordinationcited by 12
  7. 2024Policy Space Response Oracles: A Surveycited by 10
  8. 2022Feasible Adversarial Robust Reinforcement Learning for Underspecified Environmentscited by 4
  9. 2023JiangJun: Mastering Xiangqi by Tackling Non-Transitivity in Two-Player Zero-Sum Gamescited by 2
  10. 2023ApproxED: Approximate exploitability descent via learned best responsescited by 0
  11. 2023Game-Theoretic Robust Reinforcement Learning Handles Temporally-Coupled Perturbationscited by 0