# Strategic Attentive Writer for Learning Macro-Actions

arXiv: https://arxiv.org/abs/1606.04695 · HTML: https://mlsanity.com/paper/1606.04695/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2016.

## Built on by (most cited first, 30 of 30)

- [Reinforcement Learning with Action Chunking](https://mlsanity.com/paper/2507.07969/) (2025)
- [Discovery of Options via Meta-Learned Subgoals](https://mlsanity.com/paper/2102.06741/) (2021)
- [TempoRL: Learning When to Act](https://mlsanity.com/paper/2106.05262/) (2021)
- [Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2505.12737/) (2025)
- [Intelligent problem-solving as integrated hierarchical reinforcement learning](https://mlsanity.com/paper/2208.08731/) (2022)
- [DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2210.05150/) (2022)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions](https://mlsanity.com/paper/2410.02743/) (2024)
- [Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration](https://mlsanity.com/paper/2410.18076/) (2024)
- [Exploring the limits of Hierarchical World Models in Reinforcement Learning](https://mlsanity.com/paper/2406.00483/) (2024)
- [Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning](https://mlsanity.com/paper/2201.09765/) (2022)
- [When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL](https://mlsanity.com/paper/2406.01163/) (2024)
- [Context-Specific Representation Abstraction for Deep Option Learning](https://mlsanity.com/paper/2109.09876/) (2021)
- [Neural Attention Models in Deep Learning: Survey and Taxonomy](https://mlsanity.com/paper/2112.05909/) (2021)
- [Planning Transformer: Long-Horizon Offline Reinforcement Learning with Planning Tokens](https://mlsanity.com/paper/2409.09513/) (2024)
- [Online Baum-Welch algorithm for Hierarchical Imitation Learning](https://mlsanity.com/paper/2103.12197/) (2021)
- [Teacher-student curriculum learning for reinforcement learning](https://mlsanity.com/paper/2210.17368/) (2022)
- [OptionZero: Planning with Learned Options](https://mlsanity.com/paper/2502.16634/) (2025)
- [Quantum Hierarchical Reinforcement Learning via Variational Quantum Circuits](https://mlsanity.com/paper/2605.03434/) (2026)
- [Graph Neural Network based Agent in Google Research Football](https://mlsanity.com/paper/2204.11142/) (2022)
- [Efficient Open-world Reinforcement Learning via Knowledge Distillation and Autonomous Rule Discovery](https://mlsanity.com/paper/2311.14270/) (2023)
- [Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2605.05544/) (2026)
- [Interpretable Reinforcement Learning Inspired by Piaget's Theory of Cognitive Development](https://mlsanity.com/paper/2102.00572/) (2021)
- [Learning Routines for Effective Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2106.02943/) (2021)
- [Hierarchical Reinforcement Learning with Optimal Level Synchronization Based on Flow-Based Deep Generative Model](https://mlsanity.com/paper/2107.08183/) (2021)
- [SHIRO: Soft Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2212.12786/) (2022)
- [Select before Act: Spatially Decoupled Action Repetition for Continuous Control](https://mlsanity.com/paper/2502.06919/) (2025)
- [MRS: Multi-Resolution Skills for HRL Agents](https://mlsanity.com/paper/2505.21410/) (2025)
- [Horizon Adaptive Offline Policy Learning via Value Stitching](https://mlsanity.com/paper/2606.21136/) (2026)
- [Learning to Play: Reinforcement Learning and Games](https://mlsanity.com/paper/2610.17391/) (2026)

## Abstract (arXiv)

We present a novel deep recurrent neural network architecture that learns to build implicit plans in an end-to-end manner by purely interacting with an environment in reinforcement learning setting. The network builds an internal plan, which is continuously updated upon observation of the next input from the environment. It can also partition this internal representation into contiguous sub- sequences by learning for how long the plan can be committed to - i.e. followed without re-planing. Combining these properties, the proposed model, dubbed STRategic Attentive Writer (STRAW) can learn high-level, temporally abstracted macro- actions of varying lengths that are solely learnt from data without any prior information. These macro-actions enable both structured exploration and economic computation. We experimentally demonstrate that STRAW delivers strong improvements on several ATARI games by employing temporally extended planning strategies (e.g. Ms. Pacman and Frostbite). It is at the same time a general algorithm that can be applied on any sequence data. To that end, we also show that when trained on text prediction task, STRAW naturally predicts frequent n-grams (instead of macro-actions), demonstrating the generality of the approach.