mlsanityData to 14 Oct 2026

Reevaluating Policy Gradient Methods for Imperfect-Information Games

arXiv 2502.08938 ↗First posted 13 Feb 2025cs.LGcited by 13

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

In the past decade, motivated by the putative failure of naive self-play deep reinforcement learning (DRL) in adversarial imperfect-information games, researchers have developed numerous DRL algorithms based on fictitious play (FP), double oracle (DO), and counterfactual regret minimization (CFR). In light of recent results of the magnetic mirror descent algorithm, we hypothesize that simpler generic policy gradient methods like PPO are competitive with or superior to these FP-, DO-, and CFR-based DRL approaches. To facilitate the resolution of this hypothesis, we implement and release the first broadly accessible exact exploitability computations for five large games. Using these games, we conduct the largest-ever exploitability comparison of DRL algorithms for imperfect-information games. Over 7000 training runs, we find that FP-, DO-, and CFR-based approaches fail to outperform generic policy gradient methods. Code is available at https://github.com/nathanlct/IIG-RL-Benchmark and https://github.com/gabrfarina/exp-a-spiel .

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 43 of its 85 references that are in the map. First posted on arXiv 13 Feb 2025.

Builds on

43 of 85 references.

References · 43

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  5. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  6. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 268
  7. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 186
  8. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 158
  9. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 143
  10. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 131
  11. 2020Phasic Policy Gradientcited by 123
  12. 2020Combining Deep Reinforcement Learning and Search for Imperfect-Information Gamescited by 114
31 more
  1. 2014Solving Large Imperfect Information Games Using CFR+cited by 78
  2. 2018Solving Imperfect-Information Games via Discounted Regret Minimizationcited by 68
  3. 2022A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Gamescited by 66
  4. 2020From Poincar'e Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularizationcited by 61
  5. 2020Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Gamescited by 47
  6. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 41
  7. 2020Revisiting Design Choices in Proximal Policy Optimizationcited by 38
  8. 2020DREAM: Deep Regret minimization with Advantage baselines and Model-free learningcited by 36
  9. 2021XDO: A Double Oracle Algorithm for Extensive-Form Gamescited by 34
  10. 2024Steering Language Models with Game-Theoretic Solverscited by 28
  11. 2019Vortices Instead of Equilibria in MinMax Optimization: Chaos and Butterfly Effects of Online Learning in Zero-Sum Gamescited by 25
  12. 2022ESCHER: Eschewing Importance Sampling in Games by Computing a History Value Function to Estimate Regretcited by 24
  13. 2019Improved Robustness and Safety for Autonomous Vehicle Control with Adversarial Reinforcement Learningcited by 23
  14. 2019Stable-Predictive Optimistic Counterfactual Regret Minimizationcited by 21
  15. 2021Online Double Oraclecited by 18
  16. 2020The Advantage Regret-Matching Actor-Criticcited by 17
  17. 2023Evolving Diverse Red-team Language Models in Multi-round Multi-agent Gamescited by 17
  18. 2023Policy Space Diversity for Non-Transitive Gamescited by 15
  19. 2019Neural Replicator Dynamicscited by 13
  20. 2021Iterative Empirical Game Solving via Single Policy Best Responsecited by 12
  21. 2022Self-Play PSRO: Toward Optimal Populations in Two-Player Zero-Sum Gamescited by 11
  22. 2023Scalable Learning of Intrusion Responses through Recursive Decompositioncited by 8
  23. 2021Solving Large-Scale Extensive-Form Network Security Games via Neural Fictitious Self-Playcited by 7
  24. 2024Grasper: A Generalist Pursuer for Pursuit-Evasion Problemscited by 6
  25. 2023Regret-Minimizing Double Oracle for Extensive-Form Gamescited by 4
  26. 2024Exponential Lower Bounds on the Double Oracle Algorithm in Zero-Sum Gamescited by 3
  27. 2022Counterfactual Regret Minimization for Anti-jamming Game of Frequency Agile Radarcited by 2
  28. 2024Fusion-PSRO: Nash Policy Fusion for Policy Space Response Oraclescited by 2
  29. 2019Improving Fictitious Play Reinforcement Learning with Expanding Modelscited by 1
  30. 2024Bridging the Gap between Discrete Agent Strategies in Game Theory and Continuous Motion Planning in Dynamic Environmentscited by 1
  31. 2024Self-adaptive PSRO: Towards an Automatic Population-based Game Solvercited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 10.

13 papers.

Sort:

By area:

  1. 2025Human-Level Competitive Pok'emon via Scalable Offline Reinforcement Learning with Transformerscited by 4
  2. 2025High entropy leads to symmetry-equivariant policies in Dec-POMDPscited by 3
  3. 2025NashPG: A Policy Gradient Method with Iteratively Refined Regularization for Finding Nash Equilibriacited by 2
  4. 2026Data-Augmented Game Starts for Accelerating Self-Play Exploration in Imperfect Information Gamescited by 1
  5. 2026EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Gamescited by 1
  6. 2026How Much Due Diligence Before You Bid? Learning in Intractable Takeover Auctionscited by 1
  7. 2026GAE Falls Short in Imperfect-Information Self-Play Reinforcement Learningcited by 0
  8. 2026Superhuman AI for Generals.io Using Self-Play Reinforcement Learningcited by 0
  9. 2026Towards Learning Representations of Policies in Two-Player Zero-Sum Imperfect-Information Gamescited by 0
  10. 2026FootsiesGym: A Fighting Game Benchmark for Two-Player Zero-Sum Imperfect-Information Gamescited by 0
  11. 2026Beyond Bayesian Nash: Learning Minimax-Regret Equilibria for Adversarial Team Games under Asymmetric Informationcited by 0
  12. 2026Does a Toehold Make a Bidder Bolder? Preemption and Multiplicity in Multi-Round Takeover Auctionscited by 0
1 more
  1. 2026NashDreamer: Model-Based Reinforcement Learning for Zero-Sum Imperfect-Information Gamescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Self-Play Reinforcement Learning under Imperfect Information in Big 2
  2. 2024A Policy-Gradient Approach to Solving Imperfect-Information Games with Best-Iterate Convergence
  3. 2025A Practical Introduction to Deep Reinforcement Learning
  4. 2026Test-time Reinforcement Learning in Imperfect Information Games
  5. 2026Revisiting Regularized Policy Optimization for Stable and Efficient Reinforcement Learning in Two-Player Games
  6. 2025Robust and Diverse Multi-Agent Learning via Rational Policy Gradient
  7. 2024A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
  8. 2024RL-CFR: Improving Action Abstraction for Imperfect Information Extensive-Form Games with Reinforcement Learning