# Universal Successor Features for Transfer Reinforcement Learning

arXiv: https://arxiv.org/abs/2001.04025 · HTML: https://mlsanity.com/paper/2001.04025/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jan 2020.

## Built on by (most cited first, 20 of 20)

- [Foundation Policies with Hilbert Representations](https://mlsanity.com/paper/2402.15567/) (2024)
- [Does Zero-Shot Reinforcement Learning Exist?](https://mlsanity.com/paper/2209.14935/) (2022)
- [Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learning](https://mlsanity.com/paper/2310.03718/) (2023)
- [A First-Occupancy Representation for Reinforcement Learning](https://mlsanity.com/paper/2109.13863/) (2021)
- [A New Representation of Successor Features for Transfer across Dissimilar Environments](https://mlsanity.com/paper/2107.08426/) (2021)
- [Learning Successor Features the Simple Way](https://mlsanity.com/paper/2410.22133/) (2024)
- [Efficient Multi-Task and Transfer Reinforcement Learning with Parameter-Compositional Framework](https://mlsanity.com/paper/2306.01839/) (2023)
- [Successor Feature Representations](https://mlsanity.com/paper/2110.15701/) (2021)
- [L2Explorer: A Lifelong Reinforcement Learning Assessment Environment](https://mlsanity.com/paper/2203.07454/) (2022)
- [Optimistic Task Inference for Behavior Foundation Models](https://mlsanity.com/paper/2510.20264/) (2025)
- [Successor Feature Neural Episodic Control](https://mlsanity.com/paper/2111.03110/) (2021)
- [Safety-Constrained Policy Transfer with Successor Features](https://mlsanity.com/paper/2211.05361/) (2022)
- [QuietPaw: Learning Quadrupedal Locomotion with Versatile Noise Preference Alignment](https://mlsanity.com/paper/2503.05035/) (2025)
- [Learn Dynamic-Aware State Embedding for Transfer Learning](https://mlsanity.com/paper/2101.02230/) (2021)
- [Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning](https://mlsanity.com/paper/2310.10818/) (2023)
- [Decoupling Exploration and Exploitation for Unsupervised Pre-training with Successor Features](https://mlsanity.com/paper/2405.02569/) (2024)
- [A Generalized Bootstrap Target for Value-Learning, Efficiently Combining Value and Feature Predictions](https://mlsanity.com/paper/2201.01836/) (2022)
- [Stacked Universal Successor Feature Approximators for Safety in Reinforcement Learning](https://mlsanity.com/paper/2409.04641/) (2024)
- [Squeezing More from the Stream : Learning Representation Online for Streaming Reinforcement Learning](https://mlsanity.com/paper/2602.09396/) (2026)
- [Robust Successor Features](https://mlsanity.com/paper/2609.31016/) (2026)

## Abstract (arXiv)

Transfer in Reinforcement Learning (RL) refers to the idea of applying knowledge gained from previous tasks to solving related tasks. Learning a universal value function (Schaul et al., 2015), which generalizes over goals and states, has previously been shown to be useful for transfer. However, successor features are believed to be more suitable than values for transfer (Dayan, 1993; Barreto et al.,2017), even though they cannot directly generalize to new goals. In this paper, we propose (1) Universal Successor Features (USFs) to capture the underlying dynamics of the environment while allowing generalization to unseen goals and (2) a flexible end-to-end model of USFs that can be trained by interacting with the environment. We show that learning USFs is compatible with any RL algorithm that learns state values using a temporal difference method. Our experiments in a simple gridworld and with two MuJoCo environments show that USFs can greatly accelerate training when learning multiple tasks and can effectively transfer knowledge to new tasks.