# Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking

arXiv: https://arxiv.org/abs/2406.03704 · HTML: https://mlsanity.com/paper/2406.03704/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 16 of its 45 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 16 of 16)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [A Closer Look at Invalid Action Masking in Policy Gradient Algorithms](https://mlsanity.com/paper/2006.14171/) (2020)
- [Mastering Complex Control in MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1912.09729/) (2019)
- [A Survey of Deep Reinforcement Learning in Video Games](https://mlsanity.com/paper/1912.10944/) (2019)
- [Continuous-Discrete Reinforcement Learning for Hybrid Control in Robotics](https://mlsanity.com/paper/2001.00449/) (2020)
- [Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking](https://mlsanity.com/paper/2205.06750/) (2022)
- [Computationally Efficient Safe Reinforcement Learning for Power Systems](https://mlsanity.com/paper/2110.10333/) (2021)
- [Linear Encodings for Polytope Containment Problems](https://mlsanity.com/paper/1903.05214/) (2019)
- [No Prior Mask: Eliminate Redundant Action for Deep Reinforcement Learning](https://mlsanity.com/paper/2312.06258/) (2023)
- [Invariant, Viability and Discriminating Kernel Under-Approximation via Zonotope Scaling](https://mlsanity.com/paper/1901.01006/) (2019)
- [Scalable Safety-Preserving Robust Control Synthesis for Continuous-Time Linear Systems](https://mlsanity.com/paper/1312.3399/) (2013)

## Built on by (most cited first, 19 of 19)

- [Directed Exploration in Reinforcement Learning from Linear Temporal Logic](https://mlsanity.com/paper/2408.09495/) (2024)
- [Action Mapping for Reinforcement Learning in Continuous Environments with Constraints](https://mlsanity.com/paper/2412.04327/) (2024)
- [Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?](https://mlsanity.com/paper/2509.12833/) (2025)
- [Increasing Interaction Fidelity: Training Routines for Biomechanical Models in HCI](https://mlsanity.com/paper/2508.16581/) (2025)
- [CAMEL: Continuous Action Masking Enabled by Large Language Models for Reinforcement Learning](https://mlsanity.com/paper/2502.11896/) (2025)
- [Embodied Escaping: End-to-End Reinforcement Learning for Robot Navigation in Narrow Environment](https://mlsanity.com/paper/2503.03208/) (2025)
- [Unifying Agent Interaction and World Information for Multi-agent Coordination](https://mlsanity.com/paper/2509.25550/) (2025)
- [Improving Stochastic Action-Constrained Reinforcement Learning via Truncated Distributions](https://mlsanity.com/paper/2511.22406/) (2025)
- [X-Safe: Embodiment-Agnostic Safeguarding for Robot Arm Manipulation Policies via Action Masking](https://mlsanity.com/paper/2606.22278/) (2026)
- [Leveraging Analytic Gradients in Provably Safe Reinforcement Learning](https://mlsanity.com/paper/2506.01665/) (2025)
- [Action Space Reduction Strategies for Reinforcement Learning in Autonomous Driving](https://mlsanity.com/paper/2507.05251/) (2025)
- [Neuro-symbolic Action Masking for Deep Reinforcement Learning](https://mlsanity.com/paper/2602.10598/) (2026)
- [Overcoming Valid Action Suppression in Unmasked Policy Gradient Algorithms](https://mlsanity.com/paper/2603.09090/) (2026)
- [Beyond Fixed Morphologies: Learning Graph Policies with Trust Region Compensation in Variable Action Spaces](https://mlsanity.com/paper/2508.14102/) (2025)
- [Balancing Specialization and Centralization: A Multi-Agent Reinforcement Learning Benchmark for Sequential Industrial Control](https://mlsanity.com/paper/2510.20408/) (2025)
- [Guardian: Decoupling Exploration from Safety in Reinforcement Learning](https://mlsanity.com/paper/2510.22859/) (2025)
- [No Free Checker: A Survey of Verifiers for Robot Policies](https://mlsanity.com/paper/2609.09250/) (2026)
- [The Geometry of Randomized Smoothing on Feasible Sets](https://mlsanity.com/paper/2609.39497/) (2026)
- [Time Window Optimization for Motor Imagery Classification via Temporal Structure Embedded Reinforcement Learning](https://mlsanity.com/paper/2610.16921/) (2026)

## Abstract (arXiv)

Continuous action spaces in reinforcement learning (RL) are commonly defined as multidimensional intervals. While intervals usually reflect the action boundaries for tasks well, they can be challenging for learning because the typically large global action space leads to frequent exploration of irrelevant actions. Yet, little task knowledge can be sufficient to identify significantly smaller state-specific sets of relevant actions. Focusing learning on these relevant actions can significantly improve training efficiency and effectiveness. In this paper, we propose to focus learning on the set of relevant actions and introduce three continuous action masking methods for exactly mapping the action space to the state-dependent set of relevant actions. Thus, our methods ensure that only relevant actions are executed, enhancing the predictability of the RL agent and enabling its use in safety-critical applications. We further derive the implications of the proposed methods on the policy gradient. Using proximal policy optimization (PPO), we evaluate our methods on four control tasks, where the relevant action set is computed based on the system dynamics and a relevant state set. Our experiments show that the three action masking methods achieve higher final rewards and converge faster than the baseline without action masking.