mlsanityData to 14 Oct 2026

A Structural Threshold in Decision Capacity Governs Collapse in Self-Play Reinforcement Learning

arXiv 2605.16315 ↗First posted 4 May 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL for games and combinatorial optimization

Abstract (arXiv)

We show that a threshold in decision capacity determines whether self-play reinforcement learning agents collapse under asymmetric rule perturbations. Across poker variants, matrix games, a dice game, and multiple learning algorithms, eliminating all positive-reach contingent decisions causes rapid convergence to a deterministic exploitation attractor, a fixed point at near-maximal loss. Preserving even a single positive-reach contingent decision point prevents this collapse. A frozen baseline and fixed-opponent control confirm that the mechanism is co-adaptation under constraint, not the perturbation itself. The phenomenon is timing-invariant, fully reversible upon action restoration, and intensifies under function approximation. These results establish a sharp threshold at zero reach-weighted contingent action capacity, with severity scaling continuously via reach-weighted capacity in the tested domains.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods. Builds on 12 of its 23 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

12 of 23 references.

References · 12

  1. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  2. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 268
  3. 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 233
  4. 2017Learning with Opponent-Learning Awarenesscited by 222
  5. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 214
  6. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 186
  7. 2020A Closer Look at Invalid Action Masking in Policy Gradient Algorithmscited by 171
  8. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 158
  9. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  10. 2019A Generalized Training Approach for Multiagent Learningcited by 54
  11. 2018Stable Opponent Shaping in Differentiable Gamescited by 49
  12. 2020Approximate exploitability: Learning a best response in large gamescited by 23

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RL
  2. 2026When Actions Disappear: Adversarial Action Removal in Self-Play Reinforcement Learning
  3. 2026Zero Collapse: A Failure Mode of Policy Gradient Methods in Discontinuous Reward Environments
  4. 2026Expected Return Causes Outcome-Level Mode Collapse in Reinforcement Learning and How to Fix It with Inverse Probability Scaling
  5. 2024A Survey on Self-play Methods in Reinforcement Learning
  6. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback
  7. 2026Implicit Strategic Optimization: Rethinking Long-Horizon Decision-Making in Adversarial Poker Environments
  8. 2026MEMO: Memory-Augmented Model Context Optimization for Robust Multi-Turn Multi-Agent LLM Games