# Successor Feature Representations

arXiv: https://arxiv.org/abs/2110.15701 · HTML: https://mlsanity.com/paper/2110.15701/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 42 references that are in the map.
- First posted on arXiv 29 Oct 2021.

## Builds on (most cited first, 20 of 20)

- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [Fast Task Inference with Variational Intrinsic Successor Features](https://mlsanity.com/paper/1906.05030/) (2019)
- [Learning One Representation to Optimize All Rewards](https://mlsanity.com/paper/2103.07945/) (2021)
- [Count-Based Exploration with the Successor Representation](https://mlsanity.com/paper/1807.11622/) (2018)
- [Universal Successor Features Approximators](https://mlsanity.com/paper/1812.07626/) (2018)
- [Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement](https://mlsanity.com/paper/1901.10964/) (2019)
- [Deep Successor Reinforcement Learning](https://mlsanity.com/paper/1606.02396/) (2016)
- [The Option Keyboard: Combining Skills in Reinforcement Learning](https://mlsanity.com/paper/2106.13105/) (2021)
- [Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments](https://mlsanity.com/paper/1612.05533/) (2016)
- [Generative Temporal Difference Learning for Infinite-Horizon Prediction](https://mlsanity.com/paper/2010.14496/) (2020)
- [Universal Successor Features for Transfer Reinforcement Learning](https://mlsanity.com/paper/2001.04025/) (2020)
- [Reinforcement Learning Approaches in Social Robotics](https://mlsanity.com/paper/2009.09689/) (2020)
- [A neurally plausible model learns successor representations in partially observable environments](https://mlsanity.com/paper/1906.09480/) (2019)
- [Successor Features Combine Elements of Model-Free and Model-based Reinforcement Learning](https://mlsanity.com/paper/1901.11437/) (2019)
- [Successor Feature Sets: Generalizing Successor Representations Across Policies](https://mlsanity.com/paper/2103.02650/) (2021)
- [Better transfer learning with inferred successor maps](https://mlsanity.com/paper/1906.07663/) (2019)
- [Foresee then Evaluate: Decomposing Value Estimation with Latent Future Prediction](https://mlsanity.com/paper/2103.02225/) (2021)
- [State2vec: Off-Policy Successor Features Approximators](https://mlsanity.com/paper/1910.10277/) (2019)

## Built on by (most cited first, 6 of 6)

- [Learning Successor Features the Simple Way](https://mlsanity.com/paper/2410.22133/) (2024)
- [Beyond Optimism: Exploration With Partially Observable Rewards](https://mlsanity.com/paper/2406.13909/) (2024)
- [State Chrono Representation for Enhancing Generalization in Reinforcement Learning](https://mlsanity.com/paper/2411.06174/) (2024)
- [π2vec: Policy Representations with Successor Features](https://mlsanity.com/paper/2306.09800/) (2023)
- [Full-Gradient Successor Feature Representations](https://mlsanity.com/paper/2604.00686/) (2026)
- [Do as the Romans Do: Learning Universal Behaviors from Heterogeneous Agents](https://mlsanity.com/paper/2606.18537/) (2026)

## Abstract (arXiv)

Transfer in Reinforcement Learning aims to improve learning performance on target tasks using knowledge from experienced source tasks. Successor Representations (SR) and their extension Successor Features (SF) are prominent transfer mechanisms in domains where reward functions change between tasks. They reevaluate the expected return of previously learned policies in a new target task to transfer their knowledge. The SF framework extended SR by linearly decomposing rewards into successor features and a reward weight vector allowing their application in high-dimensional tasks. But this came with the cost of having a linear relationship between reward functions and successor features, limiting its application to tasks where such a linear relationship exists. We propose a novel formulation of SR based on learning the cumulative discounted probability of successor features, called Successor Feature Representations (SFR). Crucially, SFR allows to reevaluate the expected return of policies for general reward functions. We introduce different SFR variations, prove its convergence, and provide a guarantee on its transfer performance. Experimental evaluations based on SFR with function approximation demonstrate its advantage over SF not only for general reward functions, but also in the case of linearly decomposable reward functions.