# Composing Task Knowledge with Modular Successor Feature Approximators

arXiv: https://arxiv.org/abs/2301.12305 · HTML: https://mlsanity.com/paper/2301.12305/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 28 Jan 2023.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Stabilizing Transformers for Reinforcement Learning](https://mlsanity.com/paper/1910.06764/) (2019)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [Recurrent Independent Mechanisms](https://mlsanity.com/paper/1909.10893/) (2019)
- [Acme: A Research Framework for Distributed Reinforcement Learning](https://mlsanity.com/paper/2006.00979/) (2020)
- [Fast Task Inference with Variational Intrinsic Successor Features](https://mlsanity.com/paper/1906.05030/) (2019)
- [Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer](https://mlsanity.com/paper/1609.07088/) (2016)
- [One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control](https://mlsanity.com/paper/2007.04976/) (2020)
- [Count-Based Exploration with the Successor Representation](https://mlsanity.com/paper/1807.11622/) (2018)
- [Universal Successor Features Approximators](https://mlsanity.com/paper/1812.07626/) (2018)
- [Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck](https://mlsanity.com/paper/1910.12911/) (2019)
- [Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning](https://mlsanity.com/paper/1706.05064/) (2017)
- [MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research](https://mlsanity.com/paper/2109.13202/) (2021)
- [Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement](https://mlsanity.com/paper/1901.10964/) (2019)
- [Composable Deep Reinforcement Learning for Robotic Manipulation](https://mlsanity.com/paper/1803.06773/) (2018)
- [The Option Keyboard: Combining Skills in Reinforcement Learning](https://mlsanity.com/paper/2106.13105/) (2021)
- [Relational recurrent neural networks](https://mlsanity.com/paper/1806.01822/) (2018)
- [Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments](https://mlsanity.com/paper/1612.05533/) (2016)
- [Visual Semantic Planning using Deep Successor Representations](https://mlsanity.com/paper/1705.08080/) (2017)
- [Discovery of Useful Questions as Auxiliary Tasks](https://mlsanity.com/paper/1909.04607/) (2019)
- [Fast and Slow Learning of Recurrent Independent Mechanisms](https://mlsanity.com/paper/2105.08710/) (2021)
- [Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learning](https://mlsanity.com/paper/1810.06530/) (2018)
- [Procedural Generalization by Planning with Self-Supervised World Models](https://mlsanity.com/paper/2111.01587/) (2021)
- [Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies](https://mlsanity.com/paper/2001.00248/) (2020)
- [PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning](https://mlsanity.com/paper/2102.12560/) (2021)
- [A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation](https://mlsanity.com/paper/2106.06854/) (2021)

## Built on by (most cited first, 7 of 7)

- [Combining Behaviors with the Successor Features Keyboard](https://mlsanity.com/paper/2310.15940/) (2023)
- [Learning Successor Features the Simple Way](https://mlsanity.com/paper/2410.22133/) (2024)
- [Predictive representations: building blocks of intelligence](https://mlsanity.com/paper/2402.06590/) (2024)
- [Constructing an Optimal Behavior Basis for the Option Keyboard](https://mlsanity.com/paper/2505.00787/) (2025)
- [Modularity Trumps Invariance for Compositional Robustness](https://mlsanity.com/paper/2306.09005/) (2023)
- [A Unified Theory of Compositionality, Modularity, and Interpretability in Markov Decision Processes](https://mlsanity.com/paper/2506.09499/) (2025)
- [Task diversity produces systematic transfer but inhibits continual reinforcement learning](https://mlsanity.com/paper/2606.00880/) (2026)

## Abstract (arXiv)

Recently, the Successor Features and Generalized Policy Improvement (SF&GPI) framework has been proposed as a method for learning, composing, and transferring predictive knowledge and behavior. SF&GPI works by having an agent learn predictive representations (SFs) that can be combined for transfer to new tasks with GPI. However, to be effective this approach requires state features that are useful to predict, and these state-features are typically hand-designed. In this work, we present a novel neural network architecture, "Modular Successor Feature Approximators" (MSFA), where modules both discover what is useful to predict, and learn their own predictive representations. We show that MSFA is able to better generalize compared to baseline architectures for learning SFs and modular architectures