# Successor Feature Sets: Generalizing Successor Representations Across Policies

arXiv: https://arxiv.org/abs/2103.02650 · HTML: https://mlsanity.com/paper/2103.02650/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 3 Mar 2021.

## Builds on (most cited first, 12 of 12)

- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Fast Task Inference with Variational Intrinsic Successor Features](https://mlsanity.com/paper/1906.05030/) (2019)
- [Universal Successor Features Approximators](https://mlsanity.com/paper/1812.07626/) (2018)
- [Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement](https://mlsanity.com/paper/1901.10964/) (2019)
- [Eigenoption Discovery through the Deep Successor Representation](https://mlsanity.com/paper/1710.11089/) (2017)
- [Deep Successor Reinforcement Learning](https://mlsanity.com/paper/1606.02396/) (2016)
- [Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments](https://mlsanity.com/paper/1612.05533/) (2016)
- [Visual Semantic Planning using Deep Successor Representations](https://mlsanity.com/paper/1705.08080/) (2017)
- [Truly Batch Apprenticeship Learning with Deep Successor Features](https://mlsanity.com/paper/1903.10077/) (2019)
- [A neurally plausible model learns successor representations in partially observable environments](https://mlsanity.com/paper/1906.09480/) (2019)
- [Universal Successor Representations for Transfer Reinforcement Learning](https://mlsanity.com/paper/1804.03758/) (2018)
- [A New Framework for Query Efficient Active Imitation Learning](https://mlsanity.com/paper/1912.13037/) (2019)

## Built on by (most cited first, 6 of 6)

- [Successor Feature Representations](https://mlsanity.com/paper/2110.15701/) (2021)
- [Successor Feature Neural Episodic Control](https://mlsanity.com/paper/2111.03110/) (2021)
- [SR-Reward: Taking The Path More Traveled](https://mlsanity.com/paper/2501.02330/) (2025)
- [π2vec: Policy Representations with Successor Features](https://mlsanity.com/paper/2306.09800/) (2023)
- [Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning](https://mlsanity.com/paper/2310.10818/) (2023)
- [A Generalized Bootstrap Target for Value-Learning, Efficiently Combining Value and Feature Predictions](https://mlsanity.com/paper/2201.01836/) (2022)

## Abstract (arXiv)

Successor-style representations have many advantages for reinforcement learning: for example, they can help an agent generalize from past experience to new goals, and they have been proposed as explanations of behavioral and neural data from human and animal learners. They also form a natural bridge between model-based and model-free RL methods: like the former they make predictions about future experiences, and like the latter they allow efficient prediction of total discounted rewards. However, successor-style representations are not optimized to generalize across policies: typically, we maintain a limited-length list of policies, and share information among them by representation learning or GPI. Successor-style representations also typically make no provision for gathering information or reasoning about latent variables. To address these limitations, we bring together ideas from predictive state representations, belief space value iteration, successor features, and convex analysis: we develop a new, general successor-style representation, together with a Bellman equation that connects multiple sources of information within this representation, including different latent states, policies, and reward functions. The new representation is highly expressive: for example, it lets us efficiently read off an optimal policy for a new reward function, or a policy that imitates a new demonstration. For this paper, we focus on exact computation of the new representation in small, known environments, since even this restricted setting offers plenty of interesting questions. Our implementation does not scale to large, unknown environments -- nor would we expect it to, since it generalizes POMDP value iteration, which is difficult to scale. However, we believe that future work will allow us to extend our ideas to approximate reasoning in large, unknown environments.