# Context-Specific Representation Abstraction for Deep Option Learning

arXiv: https://arxiv.org/abs/2109.09876 · HTML: https://mlsanity.com/paper/2109.09876/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 20 Sep 2021.

## Builds on (most cited first, 22 of 22)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Recurrent Independent Mechanisms](https://mlsanity.com/paper/1909.10893/) (2019)
- [Deep Variational Reinforcement Learning for POMDPs](https://mlsanity.com/paper/1806.02426/) (2018)
- [Near-Optimal Representation Learning for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1810.01257/) (2018)
- [Meta Learning Shared Hierarchies](https://mlsanity.com/paper/1710.09767/) (2017)
- [Context-Specific Independence in Bayesian Networks](https://mlsanity.com/paper/1302.3562/) (2013)
- [Learning Abstract Options](https://mlsanity.com/paper/1810.11583/) (2018)
- [Strategic Attentive Writer for Learning Macro-Actions](https://mlsanity.com/paper/1606.04695/) (2016)
- [Learning Causal State Representations of Partially Observable Environments](https://mlsanity.com/paper/1906.10437/) (2019)
- [Options of Interest: Temporal Abstraction with Interest Functions](https://mlsanity.com/paper/2001.00271/) (2020)
- [Hierarchical Reinforcement Learning with Hindsight](https://mlsanity.com/paper/1805.08180/) (2018)
- [CAMPs: Learning Context-Specific Abstractions for Efficient Planning in Factored MDPs](https://mlsanity.com/paper/2007.13202/) (2020)
- [Adaptive Skills, Adaptive Partitions (ASAP)](https://mlsanity.com/paper/1602.03351/) (2016)
- [On the Role of Weight Sharing During Deep Option Learning](https://mlsanity.com/paper/1912.13408/) (2019)
- [Diversity-Enriched Option-Critic](https://mlsanity.com/paper/2011.02565/) (2020)
- [Crossmodal Attentive Skill Learner](https://mlsanity.com/paper/1711.10314/) (2017)
- [Learning Hierarchical Teaching Policies for Cooperative Agents](https://mlsanity.com/paper/1903.03216/) (2019)

## Built on by (most cited first, 8 of 8)

- [Position: Theory of Mind Benchmarks are Broken for Large Language Models](https://mlsanity.com/paper/2412.19726/) (2024)
- [Continual Learning In Environments With Polynomial Mixing Times](https://mlsanity.com/paper/2112.07066/) (2021)
- [Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference](https://mlsanity.com/paper/2412.14355/) (2024)
- [MultiPrompter: Cooperative Prompt Optimization with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2310.16730/) (2023)
- [Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environments](https://mlsanity.com/paper/2512.19154/) (2025)
- [Hierarchical Reinforcement Learning with AI Planning Models](https://mlsanity.com/paper/2203.00669/) (2022)
- [The Effectiveness of Approximate Regularized Replay for Efficient Supervised Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2512.22337/) (2025)
- [Joint Learning of Hierarchical Neural Options and Abstract World Model](https://mlsanity.com/paper/2602.02799/) (2026)

## Abstract (arXiv)

Hierarchical reinforcement learning has focused on discovering temporally extended actions, such as options, that can provide benefits in problems requiring extensive exploration. One promising approach that learns these options end-to-end is the option-critic (OC) framework. We examine and show in this paper that OC does not decompose a problem into simpler sub-problems, but instead increases the size of the search over policy space with each option considering the entire state space during learning. This issue can result in practical limitations of this method, including sample inefficient learning. To address this problem, we introduce Context-Specific Representation Abstraction for Deep Option Learning (CRADOL), a new framework that considers both temporal abstraction and context-specific representation abstraction to effectively reduce the size of the search over policy space. Specifically, our method learns a factored belief state representation that enables each option to learn a policy over only a subsection of the state space. We test our method against hierarchical, non-hierarchical, and modular recurrent neural network baselines, demonstrating significant sample efficiency improvements in challenging partially observable environments.