mlsanityData to 7 Oct 2026

A Survey on Self-play Methods in Reinforcement Learning

arXiv 2408.01072 ↗First posted 2 Aug 2024cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

Self-play, a learning paradigm where agents iteratively refine their policies by interacting with historical or concurrent versions of themselves or other evolving agents, has shown remarkable success in solving complex non-cooperative multi-agent tasks. Despite its growing prominence in multi-agent reinforcement learning (MARL), such as Go, poker, and video games, a comprehensive and structured understanding of self-play remains lacking. This survey fills this gap by offering a comprehensive roadmap to the diverse landscape of self-play methods. We begin by introducing the necessary preliminaries, including the MARL framework and basic game theory concepts. Then, it provides a unified framework and classifies existing self-play algorithms within this framework. Moreover, the paper bridges the gap between the algorithms and their practical implications by illustrating the role of self-play in different non-cooperative scenarios. Finally, the survey highlights open challenges and future research directions in self-play.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 94 of its 161 references that are in the map. First posted on arXiv 2 Aug 2024.

Builds on

94 of 161 references.

References · 94

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2015Continuous control with deep reinforcement learningcited by 3,279
  7. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  10. 2016Generative Adversarial Imitation Learningcited by 1,576
  11. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  12. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
38 more
  1. 2017Counterfactual Multi-Agent Policy Gradientscited by 733
  2. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  3. 2022Large Language Models Can Self-Improvecited by 656
  4. 2024Self-Rewarding Language Modelscited by 654
  5. 2019The StarCraft Multi-Agent Challengecited by 589
  6. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  7. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  8. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  9. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  10. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  11. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
  12. 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 228
  13. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  14. 2019MAVEN: Multi-Agent Variational Explorationcited by 181
  15. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 180
  16. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 153
  17. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 150
  18. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Gamecited by 148
  19. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 129
  20. 2017Emergent Complexity via Multi-Agent Competitioncited by 117
  21. 2020Combining Deep Reinforcement Learning and Search for Imperfect-Information Gamescited by 110
  22. 2018Deep Counterfactual Regret Minimizationcited by 102
  23. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  24. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 80
  25. 2014Solving Large Imperfect Information Games Using CFR+cited by 78
  26. 2020Towards Playing Full MOBA Games with Deep Reinforcement Learningcited by 78
  27. 2021Learning and Planning in Complex Action Spacescited by 76
  28. 2018Solving Imperfect-Information Games via Discounted Regret Minimizationcited by 68
  29. 2020Real World Games Look Like Spinning Topscited by 66
  30. 2021DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learningcited by 65
  31. 2020From Poincar'e Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularizationcited by 59
  32. 2020Suphx: Mastering Mahjong with Deep Reinforcement Learningcited by 55
  33. 2019A Generalized Training Approach for Multiagent Learningcited by 53
  34. 2019α-Rank: Multi-Agent Evaluation by Evolutioncited by 50
  35. 2020Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Gamescited by 47
  36. 2021Modelling Behavioural Diversity for Learning in Open-Ended Gamescited by 39
  37. 2020DREAM: Deep Regret minimization with Advantage baselines and Model-free learningcited by 36
  38. 2019RLCard: A Toolkit for Reinforcement Learning in Card Gamescited by 34

50 most cited of 94 All 94 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 12.

26 papers.

Sort:

By area:

  1. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  3. 2025Search Self-play: Pushing the Frontier of Agent Capability without Supervisioncited by 31
  4. 2025MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMscited by 14
  5. 2025MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Trainingcited by 12
  6. 2024The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trendscited by 11
  7. 2025Improving Rationality in the Reasoning Process of Language Models through Self-playing Gamecited by 7
  8. 2026Digital Red Queen: Adversarial Program Evolution in Core War with LLMscited by 7
  9. 2026How Far Are LLMs from Professional Poker Players? Revisiting Game-Theoretic Reasoning with Agentic Tool Usecited by 7
  10. 2025SPFT-SQL: Enhancing Large Language Model for Text-to-SQL Parsing by Self-Play Fine-Tuningcited by 5
  11. 2025Leveraging Imperfection with MEDLEY A Multi-Model Approach Harnessing Bias in Medical AIcited by 4
  12. 2025VolleyBots: A Testbed for Multi-Drone Volleyball Game Combining Motion Control and Strategic Playcited by 3
14 more
  1. 2026Adaptive Time Series Reasoning via Segment Selectioncited by 3
  2. 2026Building Better Environments for Autonomous Cyber Defencecited by 3
  3. 2024Syllabus: Portable Curricula for Reinforcement Learning Agentscited by 2
  4. 2026Game-Theoretic Lens on LLM-based Multi-Agent Systemscited by 2
  5. 2026Self-Evolving Deep Research via Joint Generation and Evaluationcited by 2
  6. 2024Optimizing Large Language Models for Dynamic Constraints through Human-in-the-Loop Discriminatorscited by 1
  7. 2026EvoNash-MARL: A Closed-Loop Multi-Agent Reinforcement Learning Framework for Medium-Horizon Equity Allocationcited by 1
  8. 2026Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Designcited by 1
  9. 2024A Case Study of Web App Coding with OpenAI Reasoning Modelscited by 0
  10. 2025PoolFlip: A Multi-Agent Reinforcement Learning Security Environment for Cyber Defensecited by 0
  11. 2026IMAGINE: Intelligent Multi-Agent Godot-based Indoor Networked Explorationcited by 0
  12. 2026NePPO: Near-Potential Policy Optimization for General-Sum Multi-Agent Reinforcement Learningcited by 0
  13. 2026Stratagem: Learning Transferable Reasoning via Trajectory-Modulated Game Self-Playcited by 0
  14. 2026AgilePE: Autonomous UAV Pursuit-Evasion via Self-Play Reinforcement Learningcited by 0