# A Closer Look at Invalid Action Masking in Policy Gradient Algorithms

arXiv: https://arxiv.org/abs/2006.14171 · HTML: https://mlsanity.com/paper/2006.14171/

## Key facts

- Cited by 171 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jun 2020.

## Built on by (most cited first, 30 of 171)

- [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://mlsanity.com/paper/2210.01241/) (2022)
- [Generating Synergistic Formulaic Alpha Collections via Reinforcement Learning](https://mlsanity.com/paper/2306.12964/) (2023)
- [Understanding Continual Learning Settings with Data Distribution Drift Analysis](https://mlsanity.com/paper/2104.01678/) (2021)
- [Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking](https://mlsanity.com/paper/2205.06750/) (2022)
- [Scalable Multi-Agent Reinforcement Learning for Warehouse Logistics with Robotic and Human Co-Workers](https://mlsanity.com/paper/2212.11498/) (2022)
- [AlphaForge: A Framework to Mine and Dynamically Combine Formulaic Alpha Factors](https://mlsanity.com/paper/2406.18394/) (2024)
- [Provably Safe Reinforcement Learning via Action Projection using Reachability Analysis and Polynomial Zonotopes](https://mlsanity.com/paper/2210.10691/) (2022)
- [Beyond CAGE: Investigating Generalization of Learned Autonomous Network Defense Policies](https://mlsanity.com/paper/2211.15557/) (2022)
- [A Reinforcement Learning Environment For Job-Shop Scheduling](https://mlsanity.com/paper/2104.03760/) (2021)
- [Gym-μRTS: Toward Affordable Full Game Real-time Strategy Games Research with Deep Reinforcement Learning](https://mlsanity.com/paper/2105.13807/) (2021)
- [Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking](https://mlsanity.com/paper/2406.03704/) (2024)
- [Learning Hierarchical Interactive Multi-Object Search for Mobile Manipulation](https://mlsanity.com/paper/2307.06125/) (2023)
- [HOPE: A Reinforcement Learning-based Hybrid Policy Path Planner for Diverse Parking Scenarios](https://mlsanity.com/paper/2405.20579/) (2024)
- [Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learning](https://mlsanity.com/paper/2311.15341/) (2023)
- [Generalising Discrete Action Spaces with Conditional Action Trees](https://mlsanity.com/paper/2104.07294/) (2021)
- [Safe and Psychologically Pleasant Traffic Signal Control with Reinforcement Learning using Action Masking](https://mlsanity.com/paper/2206.10122/) (2022)
- [Introducing PetriRL: An Innovative Framework for JSSP Resolution Integrating Petri nets and Event-based Reinforcement Learning](https://mlsanity.com/paper/2402.00046/) (2024)
- [Deep Reinforcement Learning for Constrained Field Development Optimization in Subsurface Two-phase Flow](https://mlsanity.com/paper/2104.00527/) (2021)
- [Learning to design without prior data: Discovering generalizable design strategies using deep learning and tree search](https://mlsanity.com/paper/2211.15068/) (2022)
- [Discovery of Optimal Quantum Error Correcting Codes via Reinforcement Learning](https://mlsanity.com/paper/2305.06378/) (2023)
- [Physics-Aware Combinatorial Assembly Sequence Planning using Data-free Action Masking](https://mlsanity.com/paper/2408.10162/) (2024)
- [Effective Analog ICs Floorplanning with Relational Graph Neural Networks and Reinforcement Learning](https://mlsanity.com/paper/2411.15212/) (2024)
- [Action Mapping for Reinforcement Learning in Continuous Environments with Constraints](https://mlsanity.com/paper/2412.04327/) (2024)
- [Discovering highly efficient low-weight quantum error-correcting codes with reinforcement learning](https://mlsanity.com/paper/2502.14372/) (2025)
- [Learning policies for resource allocation in business processes](https://mlsanity.com/paper/2304.09970/) (2023)
- [Reinforced Context Order Recovery for Adaptive Reasoning and Planning](https://mlsanity.com/paper/2508.13070/) (2025)
- [CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms](https://mlsanity.com/paper/2111.08819/) (2021)
- [Learning to Recharge: UAV Coverage Path Planning through Deep Reinforcement Learning](https://mlsanity.com/paper/2309.03157/) (2023)
- [Scaling Team Coordination on Graphs with Reinforcement Learning](https://mlsanity.com/paper/2403.05787/) (2024)
- [Revisiting Space Mission Planning: A Reinforcement Learning-Guided Approach for Multi-Debris Rendezvous](https://mlsanity.com/paper/2409.16882/) (2024)

## Abstract (arXiv)

In recent years, Deep Reinforcement Learning (DRL) algorithms have achieved state-of-the-art performance in many challenging strategy games. Because these games have complicated rules, an action sampled from the full discrete action distribution predicted by the learned policy is likely to be invalid according to the game rules (e.g., walking into a wall). The usual approach to deal with this problem in policy gradient algorithms is to "mask out" invalid actions and just sample from the set of valid actions. The implications of this process, however, remain under-investigated. In this paper, we 1) show theoretical justification for such a practice, 2) empirically demonstrate its importance as the space of invalid actions grows, and 3) provide further insights by evaluating different action masking regimes, such as removing masking after an agent has been trained using masking. The source code can be found at https://github.com/vwxyzjn/invalid-action-masking