# Attention Option-Critic

arXiv: https://arxiv.org/abs/2201.02628 · HTML: https://mlsanity.com/paper/2201.02628/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 7 Jan 2022.

## Builds on (most cited first, 11 of 11)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [When Waiting is not an Option : Learning Options with a Deliberation Cost](https://mlsanity.com/paper/1709.04571/) (2017)
- [Learning Abstract Options](https://mlsanity.com/paper/1810.11583/) (2018)
- [Discovering Options for Exploration by Minimizing Cover Time](https://mlsanity.com/paper/1903.00606/) (2019)
- [Options of Interest: Temporal Abstraction with Interest Functions](https://mlsanity.com/paper/2001.00271/) (2020)
- [Deep Attention Recurrent Q-Network](https://mlsanity.com/paper/1512.01693/) (2015)
- [The Termination Critic](https://mlsanity.com/paper/1902.09996/) (2019)

## Built on by (most cited first, 8 of 8)

- [SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions](https://mlsanity.com/paper/2410.18416/) (2024)
- [Option Discovery Using LLM-guided Semantic Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2503.19007/) (2025)
- [ChemGymRL: An Interactive Framework for Reinforcement Learning for Digital Chemistry](https://mlsanity.com/paper/2305.14177/) (2023)
- [Hierarchical Reinforcement Learning in Multi-Goal Spatial Navigation with Autonomous Mobile Robots](https://mlsanity.com/paper/2504.18794/) (2025)
- [Hierarchical Reinforcement Learning for Optimal Agent Grouping in Cooperative Systems](https://mlsanity.com/paper/2501.06554/) (2025)
- [Enter the Void - Planning to Seek Entropy When Reward is Scarce](https://mlsanity.com/paper/2505.16787/) (2025)
- [ToSCA: Leveraging Hierarchical Reinforcement Learning on Temporal and Strategic Abstractions of Conversational Agents](https://mlsanity.com/paper/2608.21969/) (2026)
- [When Do Options Help? Policy Necrosis and Redundant Coverage in Option-Critic](https://mlsanity.com/paper/2609.05508/) (2026)

## Abstract (arXiv)

Temporal abstraction in reinforcement learning is the ability of an agent to learn and use high-level behaviors, called options. The option-critic architecture provides a gradient-based end-to-end learning method to construct options. We propose an attention-based extension to this framework, which enables the agent to learn to focus different options on different aspects of the observation space. We show that this leads to behaviorally diverse options which are also capable of state abstraction, and prevents the degeneracy problems of option domination and frequent option switching that occur in option-critic, while achieving a similar sample complexity. We also demonstrate the more efficient, interpretable, and reusable nature of the learned options in comparison with option-critic, through different transfer learning tasks. Experimental results in a relatively simple four-rooms environment and the more complex ALE (Arcade Learning Environment) showcase the efficacy of our approach.