# Temporally Abstract Partial Models

arXiv: https://arxiv.org/abs/2108.03213 · HTML: https://mlsanity.com/paper/2108.03213/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 48 references that are in the map.
- First posted on arXiv 6 Aug 2021.

## Builds on (most cited first, 17 of 17)

- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Value Prediction Network](https://mlsanity.com/paper/1707.03497/) (2017)
- [Shaping Belief States with Generative Environment Models for RL](https://mlsanity.com/paper/1906.09237/) (2019)
- [When Waiting is not an Option : Learning Options with a Deliberation Cost](https://mlsanity.com/paper/1709.04571/) (2017)
- [COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration](https://mlsanity.com/paper/1905.09275/) (2019)
- [Self-Correcting Models for Model-Based Reinforcement Learning](https://mlsanity.com/paper/1612.06018/) (2016)
- [Deep Affordance Foresight: Planning Through What Can Be Done in the Future](https://mlsanity.com/paper/2011.08424/) (2020)
- [Neural Predictive Belief Representations](https://mlsanity.com/paper/1811.06407/) (2018)
- [What can I do here? A Theory of Affordances in Reinforcement Learning](https://mlsanity.com/paper/2006.15085/) (2020)
- [Options of Interest: Temporal Abstraction with Interest Functions](https://mlsanity.com/paper/2001.00271/) (2020)
- [A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2106.02097/) (2021)
- [The Termination Critic](https://mlsanity.com/paper/1902.09996/) (2019)
- [Lifelong Learning with a Changing Action Set](https://mlsanity.com/paper/1906.01770/) (2019)
- [Launchpad: A Programming Model for Distributed Machine Learning Research](https://mlsanity.com/paper/2106.04516/) (2021)
- [What can you do with a rock? Affordance extraction via word embeddings](https://mlsanity.com/paper/1703.03429/) (2017)
- [Learning Awareness Models](https://mlsanity.com/paper/1804.06318/) (2018)
- [Multi-modal Feedback for Affordance-driven Interactive Reinforcement Learning](https://mlsanity.com/paper/1807.09991/) (2018)

## Built on by (most cited first, 11 of 11)

- [Flexible Option Learning](https://mlsanity.com/paper/2112.03097/) (2021)
- [Empowering Large Language Models on Robotic Manipulation with Affordance Prompting](https://mlsanity.com/paper/2404.11027/) (2024)
- [Recent Advances of Deep Robotic Affordance Learning: A Reinforcement Learning Perspective](https://mlsanity.com/paper/2303.05344/) (2023)
- [The Paradox of Choice: Using Attention in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2201.09653/) (2022)
- [Minimal Value-Equivalent Partial Models for Scalable and Robust Planning in Lifelong Reinforcement Learning](https://mlsanity.com/paper/2301.10119/) (2023)
- [Self-Supervised Learning of Action Affordances as Interaction Modes](https://mlsanity.com/paper/2305.17565/) (2023)
- [Toward Human-AI Alignment in Large-Scale Multi-Player Games](https://mlsanity.com/paper/2402.03575/) (2024)
- [Joint Learning of Hierarchical Neural Options and Abstract World Model](https://mlsanity.com/paper/2602.02799/) (2026)
- [Forecaster: Towards Temporally Abstract Tree-Search Planning from Pixels](https://mlsanity.com/paper/2310.09997/) (2023)
- [Partial Models for Building Adaptive Model-Based Reinforcement Learning Agents](https://mlsanity.com/paper/2405.16899/) (2024)
- [A Unified Theory of Compositionality, Modularity, and Interpretability in Markov Decision Processes](https://mlsanity.com/paper/2506.09499/) (2025)

## Abstract (arXiv)

Humans and animals have the ability to reason and make predictions about different courses of action at many time scales. In reinforcement learning, option models (Sutton, Precup & Singh, 1999; Precup, 2000) provide the framework for this kind of temporally abstract prediction and reasoning. Natural intelligent agents are also able to focus their attention on courses of action that are relevant or feasible in a given situation, sometimes termed affordable actions. In this paper, we define a notion of affordances for options, and develop temporally abstract partial option models, that take into account the fact that an option might be affordable only in certain situations. We analyze the trade-offs between estimation and approximation error in planning and learning when using such models, and identify some interesting special cases. Additionally, we demonstrate empirically the potential impact of partial option models on the efficiency of planning.