# Policy Space Response Oracles: A Survey

arXiv: https://arxiv.org/abs/2403.02227 · HTML: https://mlsanity.com/paper/2403.02227/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 109 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 30 of 53)

- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [OpenSpiel: A Framework for Reinforcement Learning in Games](https://mlsanity.com/paper/1908.09453/) (2019)
- [Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learning](https://mlsanity.com/paper/2206.15378/) (2022)
- [Deep Counterfactual Regret Minimization](https://mlsanity.com/paper/1811.00164/) (2018)
- [Open-ended Learning in Symmetric Zero-sum Games](https://mlsanity.com/paper/1901.08106/) (2019)
- [Game-Theoretic Multiagent Reinforcement Learning](https://mlsanity.com/paper/2011.00583/) (2020)
- [Real World Games Look Like Spinning Tops](https://mlsanity.com/paper/2004.09468/) (2020)
- [A Generalized Training Approach for Multiagent Learning](https://mlsanity.com/paper/1909.12823/) (2019)
- [α-Rank: Multi-Agent Evaluation by Evolution](https://mlsanity.com/paper/1903.01373/) (2019)
- [Re-evaluating Evaluation](https://mlsanity.com/paper/1806.02643/) (2018)
- [Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Games](https://mlsanity.com/paper/2006.08555/) (2020)
- [Modelling Behavioural Diversity for Learning in Open-Ended Games](https://mlsanity.com/paper/2103.07927/) (2021)
- [XDO: A Double Oracle Algorithm for Extensive-Form Games](https://mlsanity.com/paper/2103.06426/) (2021)
- [Steering Language Models with Game-Theoretic Solvers](https://mlsanity.com/paper/2402.01704/) (2024)
- [Depth-Limited Solving for Imperfect-Information Games](https://mlsanity.com/paper/1805.08195/) (2018)
- [ASP: Learn a Universal Neural Solver!](https://mlsanity.com/paper/2303.00466/) (2023)
- [Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems](https://mlsanity.com/paper/2102.07659/) (2021)
- [ESCHER: Eschewing Importance Sampling in Games by Computing a History Value Function to Estimate Regret](https://mlsanity.com/paper/2206.04122/) (2022)
- [MALib: A Parallel Framework for Population-based Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2106.07551/) (2021)
- [Multi-Agent Training beyond Zero-Sum with Correlated Equilibrium Meta-Solvers](https://mlsanity.com/paper/2106.09435/) (2021)
- [Computational Results for Extensive-Form Adversarial Team Games](https://mlsanity.com/paper/1711.06930/) (2017)
- [Deep Reinforcement Learning for Green Security Games with Real-Time Information](https://mlsanity.com/paper/1811.02483/) (2018)
- [Online Double Oracle](https://mlsanity.com/paper/2103.07780/) (2021)
- [Policy Space Diversity for Non-Transitive Games](https://mlsanity.com/paper/2306.16884/) (2023)
- [Diversifying AI: Towards Creative Chess with AlphaZero](https://mlsanity.com/paper/2308.09175/) (2023)
- [A Game-Theoretic Approach for Improving Generalization Ability of TSP Solvers](https://mlsanity.com/paper/2110.15105/) (2021)
- [Learning Equilibria in Mean-Field Games: Introducing Mean-Field PSRO](https://mlsanity.com/paper/2111.08350/) (2021)
- [A Game-Theoretic Framework for Managing Risk in Multi-Agent Systems](https://mlsanity.com/paper/2205.15434/) (2022)
- [Finding Needles in a Moving Haystack: Prioritizing Alerts with Adversarial Reinforcement Learning](https://mlsanity.com/paper/1906.08805/) (2019)
- [Robust Reinforcement Learning Under Minimax Regret for Green Security](https://mlsanity.com/paper/2106.08413/) (2021)

## Built on by (most cited first, 10 of 10)

- [A Survey on Self-play Methods in Reinforcement Learning](https://mlsanity.com/paper/2408.01072/) (2024)
- [Faster Rates for No-Regret Learning in General Games via Cautious Optimism](https://mlsanity.com/paper/2503.24340/) (2025)
- [Discovering Multiagent Learning Algorithms with Large Language Models](https://mlsanity.com/paper/2602.16928/) (2026)
- [NashPG: A Policy Gradient Method with Iteratively Refined Regularization for Finding Nash Equilibria](https://mlsanity.com/paper/2510.18183/) (2025)
- [Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models](https://mlsanity.com/paper/2603.10098/) (2026)
- [Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learning](https://mlsanity.com/paper/2603.00374/) (2026)
- [PoolFlip: A Multi-Agent Reinforcement Learning Security Environment for Cyber Defense](https://mlsanity.com/paper/2508.19488/) (2025)
- [The Hidden Game Problem](https://mlsanity.com/paper/2510.03845/) (2025)
- [Perturbing Best Responses in Zero-Sum Games](https://mlsanity.com/paper/2511.12523/) (2025)
- [IB-RL: Isolated Bilateral Reinforcement Learning for Strategic Dialogue Agents](https://mlsanity.com/paper/2608.06735/) (2026)

## Abstract (arXiv)

Game theory provides a mathematical way to study the interaction between multiple decision makers. However, classical game-theoretic analysis is limited in scalability due to the large number of strategies, precluding direct application to more complex scenarios. This survey provides a comprehensive overview of a framework for large games, known as Policy Space Response Oracles (PSRO), which holds promise to improve scalability by focusing attention on sufficient subsets of strategies. We first motivate PSRO and provide historical context. We then focus on the strategy exploration problem for PSRO: the challenge of assembling effective subsets of strategies that still represent the original game well with minimum computational cost. We survey current research directions for enhancing the efficiency of PSRO, and explore the applications of PSRO across various domains. We conclude by discussing open questions and future research.