mlsanityData to 14 Oct 2026

When Actions Disappear: Adversarial Action Removal in Self-Play Reinforcement Learning

arXiv 2605.16312 ↗First posted 4 May 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Adversarial robustness and RL in finance

Abstract (arXiv)

We study adversarial action masking in self-play reinforcement learning: an attacker selectively removes legal actions from a victim's action set. Unlike observation or action perturbations, removal eliminates decision options before the agent acts. Across poker games scaling from 6 to 5,531 information states and two non-poker domains, learned masking causes substantially more damage than random masking and learned perturbation baselines. The attack persists across Q-learning, PPO, NFSP, neural NFSP, and DQN victims; transfers across agents; is amplified by self-play; and shows no recovery under extended masked training. Mechanistically, the adversary targets high-value decision points, captured by reach-weighted contingent action capacity (CACw) and a value-weighted refinement CACv. These results identify action availability as a distinct robustness surface in self-play RL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Adversarial robustness and RL in finance, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 25 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

15 of 25 references.

References · 15

  1. 2017Robust Adversarial Reinforcement Learningcited by 451
  2. 2017Adversarial Attacks on Neural Network Policiescited by 320
  3. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 268
  4. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 214
  5. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 186
  6. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  7. 2020A Closer Look at Invalid Action Masking in Policy Gradient Algorithmscited by 171
  8. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 158
  9. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 135
  10. 2018Deep Counterfactual Regret Minimizationcited by 103
  11. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  12. 2020Adaptive Reward-Poisoning Attacks against Reinforcement Learningcited by 86
3 more
  1. 2021Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RLcited by 55
  2. 2020On the Robustness of Cooperative Multi-Agent Reinforcement Learningcited by 39
  3. 2022Sparse Adversarial Attack in Multi-agent Reinforcement Learningcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Structural Threshold in Decision Capacity Governs Collapse in Self-Play Reinforcement Learning
  2. 2026VAM: Verbalized Action Masking for Controllable Exploration in RL Post-Training -- A Chess Case Study
  3. 2026Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning
  4. 2024SUB-PLAY: Adversarial Policies against Partially Observed Multi-Agent Reinforcement Learning Systems
  5. 2026Beware Untrusted Simulators -- Reward-Free Backdoor Attacks in Reinforcement Learning
  6. 2026Overcoming Valid Action Suppression in Unmasked Policy Gradient Algorithms
  7. 2024Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses
  8. 2025Robust In-Context Reinforcement Learning Under Reward Poisoning Attacks