mlsanityData to 7 Oct 2026

Policy Space Diversity for Non-Transitive Games

arXiv 2306.16884 ↗First posted 29 Jun 2023cs.GT cs.LG cs.MAcited by 15

Not on the map. Why?

Abstract (arXiv)

Policy-Space Response Oracles (PSRO) is an influential algorithm framework for approximating a Nash Equilibrium (NE) in multi-agent non-transitive games. Many previous studies have been trying to promote policy diversity in PSRO. A major weakness in existing diversity metrics is that a more diverse (according to their diversity metrics) population does not necessarily mean (as we proved in the paper) a better approximation to a NE. To alleviate this problem, we propose a new diversity metric, the improvement of which guarantees a better approximation to a NE. Meanwhile, we develop a practical and well-justified method to optimize our diversity metric using only state-action samples. By incorporating our diversity regularization into the best response solving in PSRO, we obtain a new PSRO variant, Policy Space Diversity PSRO (PSD-PSRO). We present the convergence property of PSD-PSRO. Empirically, extensive experiments on various games demonstrate that PSD-PSRO is more effective in producing significantly less exploitable policies than state-of-the-art PSRO variants.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 28 of its 56 references that are in the map. First posted on arXiv 29 Jun 2023.

Builds on

28 of 56 references.

References · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  5. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  6. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  7. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  8. 2014Robots that can adapt like animalscited by 203
  9. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  10. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 153
  11. 2018The Mechanics of n-Player Differentiable Gamescited by 118
  12. 2020Effective Diversity in Population Based Reinforcement Learningcited by 116
16 more
  1. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  2. 2020One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RLcited by 79
  3. 2020Real World Games Look Like Spinning Topscited by 66
  4. 2018Diversity-Driven Exploration Strategy for Deep Reinforcement Learningcited by 65
  5. 2020Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Gamescited by 47
  6. 2021Modelling Behavioural Diversity for Learning in Open-Ended Gamescited by 39
  7. 2021Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomizationcited by 35
  8. 2021Last-iterate Convergence in Extensive-Form Gamescited by 27
  9. 2022Near-Optimal Learning of Extensive-Form Games with Imperfect Informationcited by 26
  10. 2019Optimistic Regret Minimization for Extensive-Form Games via Dilated Distance-Generating Functionscited by 25
  11. 2022NeuPL: Neural Population Learningcited by 14
  12. 2020Non-local Policy Optimization via Diversity-regularized Collaborative Explorationcited by 13
  13. 2021Iterative Empirical Game Solving via Single Policy Best Responsecited by 11
  14. 2021Subgame solving without common knowledgecited by 11
  15. 2022Efficient Policy Space Response Oraclescited by 8
  16. 2021Equivalence Analysis between Counterfactual Regret Minimization and Online Mirror Descentcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 5; 2026: 5.

15 papers.

Sort:

By area:

  1. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 39
  2. 2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Playcited by 37
  3. 2024A Survey on Self-play Methods in Reinforcement Learningcited by 26
  4. 2025Reevaluating Policy Gradient Methods for Imperfect-Information Gamescited by 13
  5. 2024Policy Space Response Oracles: A Surveycited by 10
  6. 2024HM3: Hierarchical Multi-Objective Model Merging for Pretrained Modelscited by 8
  7. 2023Diversity from Human Feedbackcited by 5
  8. 2026Beyond Retrieval: Learning Compact User Representations for Scalable LLM Personalizationcited by 3
  9. 2024Fusion-PSRO: Nash Policy Fusion for Policy Space Response Oraclescited by 2
  10. 2025PoolFlip: A Multi-Agent Reinforcement Learning Security Environment for Cyber Defensecited by 0
  11. 2025Simulation-Free PSRO: Removing Game Simulation from Policy Space Response Oraclescited by 0
  12. 2026Rethinking Policy Diversity in Ensemble Policy Gradient in Large-Scale Reinforcement Learningcited by 0
3 more
  1. 2026NePPO: Near-Potential Policy Optimization for General-Sum Multi-Agent Reinforcement Learningcited by 0
  2. 2026Global Policy-Space Response Oracles for Two-Player Zero-Sum Gamescited by 0
  3. 2026Explicit Trajectory Diversity for RL-Based Post-Training of LLM Agentscited by 0