# Growing Action Spaces

arXiv: https://arxiv.org/abs/1906.12266 · HTML: https://mlsanity.com/paper/1906.12266/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2019.

## Built on by (most cited first, 29 of 29)

- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Reinforcement Learning for Adaptive Mesh Refinement](https://mlsanity.com/paper/2103.01342/) (2021)
- [Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies](https://mlsanity.com/paper/2111.02552/) (2021)
- [DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systems](https://mlsanity.com/paper/2206.00484/) (2022)
- [Solving Continuous Control via Q-learning](https://mlsanity.com/paper/2210.12566/) (2022)
- [Deep hierarchical reinforcement agents for automated penetration testing](https://mlsanity.com/paper/2109.06449/) (2021)
- [Prompt Engineering Through the Lens of Optimal Control](https://mlsanity.com/paper/2310.14201/) (2023)
- [Bingham Policy Parameterization for 3D Rotations in Reinforcement Learning](https://mlsanity.com/paper/2202.03957/) (2022)
- [Solving Large-Scale Extensive-Form Network Security Games via Neural Fictitious Self-Play](https://mlsanity.com/paper/2106.00897/) (2021)
- [Behaviour-Diverse Automatic Penetration Testing: A Curiosity-Driven Multi-Objective Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2202.10630/) (2022)
- [Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolution](https://mlsanity.com/paper/2404.04253/) (2024)
- [Stochastic Q-learning for Large Discrete Action Spaces](https://mlsanity.com/paper/2405.10310/) (2024)
- [BraVE: Offline Reinforcement Learning for Discrete Combinatorial Action Spaces](https://mlsanity.com/paper/2410.21151/) (2024)
- [An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces](https://mlsanity.com/paper/2411.11088/) (2024)
- [Online Decision Making with Generative Action Sets](https://mlsanity.com/paper/2509.25777/) (2025)
- [Cooperative Multi-Agent Reinforcement Learning for Inventory Management](https://mlsanity.com/paper/2304.08769/) (2023)
- [Forecasting Evolution of Clusters in Game Agents with Hebbian Learning](https://mlsanity.com/paper/2209.06904/) (2022)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)
- [Action Space Reduction Strategies for Reinforcement Learning in Autonomous Driving](https://mlsanity.com/paper/2507.05251/) (2025)
- [Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning](https://mlsanity.com/paper/2510.12096/) (2025)
- [Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution](https://mlsanity.com/paper/2604.03472/) (2026)
- [More Efficient Exploration with Symbolic Priors on Action Sequence Equivalences](https://mlsanity.com/paper/2110.10632/) (2021)
- [GAIL-PT: A Generic Intelligent Penetration Testing Framework with Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2204.01975/) (2022)
- [Constructing Ancestral Recombination Graphs through Reinforcement Learning](https://mlsanity.com/paper/2406.12022/) (2024)
- [Learning in complex action spaces without policy gradients](https://mlsanity.com/paper/2410.06317/) (2024)
- [SAINT: Attention-Based Policies for Discrete Combinatorial Action Spaces](https://mlsanity.com/paper/2505.12109/) (2025)
- [Where to Intervene: Action Selection in Deep Reinforcement Learning](https://mlsanity.com/paper/2507.04187/) (2025)
- [Improving and Accelerating Offline RL in Large Discrete Action Spaces with Structured Policy Initialization](https://mlsanity.com/paper/2601.04441/) (2026)

## Abstract (arXiv)

In complex tasks, such as those with large combinatorial action spaces, random exploration may be too inefficient to achieve meaningful learning progress. In this work, we use a curriculum of progressively growing action spaces to accelerate learning. We assume the environment is out of our control, but that the agent may set an internal curriculum by initially restricting its action space. Our approach uses off-policy reinforcement learning to estimate optimal value functions for multiple action spaces simultaneously and efficiently transfers data, value estimates, and state representations from restricted action spaces to the full task. We show the efficacy of our approach in proof-of-concept control tasks and on challenging large-scale StarCraft micromanagement tasks with large, multi-agent action spaces.