# Deep Reinforcement Learning With Macro-Actions

arXiv: https://arxiv.org/abs/1606.04615 · HTML: https://mlsanity.com/paper/1606.04615/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2016.

## Built on by (most cited first, 14 of 14)

- [Reinforcement Learning with Action Chunking](https://mlsanity.com/paper/2507.07969/) (2025)
- [An Analysis of Frame-skipping in Reinforcement Learning](https://mlsanity.com/paper/2102.03718/) (2021)
- [Addressing Action Oscillations through Learning Policy Inertia](https://mlsanity.com/paper/2103.02287/) (2021)
- [When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL](https://mlsanity.com/paper/2406.01163/) (2024)
- [Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces](https://mlsanity.com/paper/2509.22963/) (2025)
- [Action abstractions for amortized sampling](https://mlsanity.com/paper/2410.15184/) (2024)
- [InvestESG: A multi-agent reinforcement learning benchmark for studying climate investment as a social dilemma](https://mlsanity.com/paper/2411.09856/) (2024)
- [Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantization](https://mlsanity.com/paper/2511.15055/) (2025)
- [Reinforcement Learning for Robot Navigation with Adaptive Forward Simulation Time (AFST) in a Semi-Markov Model](https://mlsanity.com/paper/2108.06161/) (2021)
- [OptionZero: Planning with Learned Options](https://mlsanity.com/paper/2502.16634/) (2025)
- [DRL: Deep Reinforcement Learning for Intelligent Robot Control -- Concept, Literature, and Future](https://mlsanity.com/paper/2105.13806/) (2021)
- [Improving planning and MBRL with temporally-extended actions](https://mlsanity.com/paper/2505.15754/) (2025)
- [Meta-learning how to Share Credit among Macro-Actions](https://mlsanity.com/paper/2506.13690/) (2025)
- [Enhancing Human-Likeness in Reinforcement Learning Agents via Hierarchical Macro Action Quantization](https://mlsanity.com/paper/2605.30928/) (2026)

## Abstract (arXiv)

Deep reinforcement learning has been shown to be a powerful framework for learning policies from complex high-dimensional sensory inputs to actions in complex tasks, such as the Atari domain. In this paper, we explore output representation modeling in the form of temporal abstraction to improve convergence and reliability of deep reinforcement learning approaches. We concentrate on macro-actions, and evaluate these on different Atari 2600 games, where we show that they yield significant improvements in learning speed. Additionally, we show that they can even achieve better scores than DQN. We offer analysis and explanation for both convergence and final results, revealing a problem deep RL approaches have with sparse reward signals.