# Action Space Shaping in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2004.00980 · HTML: https://mlsanity.com/paper/2004.00980/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2020.

## Built on by (most cited first, 30 of 46)

- [Reinforced Neighborhood Selection Guided Multi-Relational Graph Neural Networks](https://mlsanity.com/paper/2104.07886/) (2021)
- [Gym-μRTS: Toward Affordable Full Game Real-time Strategy Games Research with Deep Reinforcement Learning](https://mlsanity.com/paper/2105.13807/) (2021)
- [Navigates Like Me: Understanding How People Evaluate Human-Like AI in Video Games](https://mlsanity.com/paper/2303.02160/) (2023)
- [Automated Reinforcement Learning: An Overview](https://mlsanity.com/paper/2201.05000/) (2022)
- [Embedding-Aligned Language Models](https://mlsanity.com/paper/2406.00024/) (2024)
- [BEDD: The MineRL BASALT Evaluation and Demonstrations Dataset for Training and Benchmarking Agents that Solve Fuzzy Tasks](https://mlsanity.com/paper/2312.02405/) (2023)
- [Learning the Optimal Power Flow: Environment Design Matters](https://mlsanity.com/paper/2403.17831/) (2024)
- [Generalising Discrete Action Spaces with Conditional Action Trees](https://mlsanity.com/paper/2104.07294/) (2021)
- [Safe and Psychologically Pleasant Traffic Signal Control with Reinforcement Learning using Action Masking](https://mlsanity.com/paper/2206.10122/) (2022)
- [RAPID: Robust and Agile Planner Using Inverse Reinforcement Learning for Vision-Based Drone Navigation](https://mlsanity.com/paper/2502.02054/) (2025)
- [The Crucial Role of Problem Formulation in Real-World Reinforcement Learning](https://mlsanity.com/paper/2503.20442/) (2025)
- [Beyond the Sum: Unlocking AI Agents Potential Through Market Forces](https://mlsanity.com/paper/2501.10388/) (2024)
- [Distilling Reinforcement Learning Tricks for Video Games](https://mlsanity.com/paper/2107.00703/) (2021)
- [Automating DBSCAN via Deep Reinforcement Learning](https://mlsanity.com/paper/2208.04537/) (2022)
- [InTune: Reinforcement Learning-based Data Pipeline Optimization for Deep Recommendation Models](https://mlsanity.com/paper/2308.08500/) (2023)
- [Learning to Recharge: UAV Coverage Path Planning through Deep Reinforcement Learning](https://mlsanity.com/paper/2309.03157/) (2023)
- [Distributional Reinforcement Learning for Scheduling of Chemical Production Processes](https://mlsanity.com/paper/2203.00636/) (2022)
- [Vision-driven UAV River Following: Benchmarking with Safe Reinforcement Learning](https://mlsanity.com/paper/2409.08511/) (2024)
- [CryoRL: Reinforcement Learning Enables Efficient Cryo-EM Data Collection](https://mlsanity.com/paper/2204.07543/) (2022)
- [Towards Modern Card Games with Large-Scale Action Spaces Through Action Representation](https://mlsanity.com/paper/2206.12700/) (2022)
- [LoopTune: Optimizing Tensor Computations with Reinforcement Learning](https://mlsanity.com/paper/2309.01825/) (2023)
- [Safety through Permissibility: Shield Construction for Fast and Safe Reinforcement Learning](https://mlsanity.com/paper/2405.19414/) (2024)
- [Reinforcement Learning-based Approach for Vehicle-to-Building Charging with Heterogeneous Agents and Long Term Rewards](https://mlsanity.com/paper/2502.18526/) (2025)
- [Adaptive and Robust DBSCAN with Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2505.04339/) (2025)
- [Adapting Vision-Language Models for Evaluating World Models](https://mlsanity.com/paper/2506.17967/) (2025)
- [Action Space Reduction Strategies for Reinforcement Learning in Autonomous Driving](https://mlsanity.com/paper/2507.05251/) (2025)
- [Neuro-symbolic Action Masking for Deep Reinforcement Learning](https://mlsanity.com/paper/2602.10598/) (2026)
- [MimicBot: Combining Imitation and Reinforcement Learning to win in Bot Bowl](https://mlsanity.com/paper/2108.09478/) (2021)
- [Context-Aware Composition of Agent Policies by Markov Decision Process Entity Embeddings and Agent Ensembles](https://mlsanity.com/paper/2308.14521/) (2023)
- [Scalable Volt-VAR Optimization using RLlib-IMPALA Framework: A Reinforcement Learning Approach](https://mlsanity.com/paper/2402.15932/) (2024)

## Abstract (arXiv)

Reinforcement learning (RL) has been successful in training agents in various learning environments, including video-games. However, such work modifies and shrinks the action space from the game's original. This is to avoid trying "pointless" actions and to ease the implementation. Currently, this is mostly done based on intuition, with little systematic research supporting the design decisions. In this work, we aim to gain insight on these action space modifications by conducting extensive experiments in video-game environments. Our results show how domain-specific removal of actions and discretization of continuous actions can be crucial for successful learning. With these insights, we hope to ease the use of RL in new environments, by clarifying what action-spaces are easy to learn.