mlsanityData to 14 Oct 2026

Data-Augmented Game Starts for Accelerating Self-Play Exploration in Imperfect Information Games

arXiv 2605.14379 ↗First posted 14 May 2026cs.LG cs.AI cs.GTcited by 1

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

Finding approximate equilibria for large-scale imperfect-information competitive games such as StarCraft, Dota, and CounterStrike remains computationally infeasible due to sparse rewards and challenging exploration over long horizons. In this paper, we propose a multi-agent starting-state sampling strategy designed to substantially accelerate online exploration in regularized policy-gradient game methods for two-player zero-sum (2p0s) games. Motivated by an assumption that offline demonstrations from skilled humans can provide good coverage of high-level strategies relevant to equilibrium play, we propose the initialization of reinforcement learning data collection at intermediate states sampled from offline data to facilitate exploration of strategically relevant subgames. Referring to this method as Data-Augmented Game Starts (DAGS), we perform experiments using synthetic datasets and analytically tractable, long-horizon control variants of two-player Kuhn Poker, Goofspiel, and a counterexample game designed to penalize biased beliefs over hidden information. Under fixed computational budgets, DAGS enables regularized policy gradient methods to achieve lower exploitability in games with significantly more challenging exploration. We show that augmenting starting state distributions when solving imperfect information games can lead to biased equilibria, and we provide a straightforward mitigation to this in the form of multi-task observation flags. Finally, we release a new set of benchmark environments that drastically increase exploration challenges and state counts in existing OpenSpiel games while keeping exploitability measurements analytically tractable.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 27 references that are in the map. First posted on arXiv 14 May 2026.

Builds on

19 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Scalable Decision Making for Games of Imperfect Information10 Nov
  2. 2025Reevaluating Policy Gradient Methods for Imperfect-Information Games13 Feb

Other references · 17

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  3. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 373
  4. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 268
  5. 2020First return, then explorecited by 266
  6. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 198
  7. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 158
  8. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 131
  9. 2022Jump-Start Reinforcement Learningcited by 112
  10. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 80
  11. 2022A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Gamescited by 66
  12. 2020Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Gamescited by 47
5 more
  1. 2020DREAM: Deep Regret minimization with Advantage baselines and Model-free learningcited by 36
  2. 2021XDO: A Double Oracle Algorithm for Extensive-Form Gamescited by 34
  3. 2022ESCHER: Eschewing Importance Sampling in Games by Computing a History Value Function to Estimate Regretcited by 24
  4. 2018Backplay: "Man muss immer umkehren"cited by 21
  5. 2022NeuPL: Neural Population Learningcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Gamescited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Self-Play Reinforcement Learning under Imperfect Information in Big 2
  2. 2026Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learning
  3. 2024A Policy-Gradient Approach to Solving Imperfect-Information Games with Best-Iterate Convergence
  4. 2024Offline Fictitious Self-Play for Competitive Games
  5. 2026MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games
  6. 2024ELA: Exploited Level Augmentation for Offline Learning in Zero-Sum Games
  7. 2026DiffNash: Finding Nash Equilibrium in High-Dimensional Continuous Action Spaces with Diffusion Policies
  8. 2026Revisiting Regularized Policy Optimization for Stable and Efficient Reinforcement Learning in Two-Player Games