# Learning Successor Features the Simple Way

arXiv: https://arxiv.org/abs/2410.22133 · HTML: https://mlsanity.com/paper/2410.22133/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 66 references that are in the map.
- First posted on arXiv 29 Oct 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks](https://mlsanity.com/paper/2306.13831/) (2023)
- [Better Fine-Tuning by Reducing Representational Collapse](https://mlsanity.com/paper/2008.03156/) (2020)
- [URLB: Unsupervised Reinforcement Learning Benchmark](https://mlsanity.com/paper/2110.15191/) (2021)
- [Fast Task Inference with Variational Intrinsic Successor Features](https://mlsanity.com/paper/1906.05030/) (2019)
- [Loss of Plasticity in Continual Deep Reinforcement Learning](https://mlsanity.com/paper/2303.07507/) (2023)
- [Universal Successor Features Approximators](https://mlsanity.com/paper/1812.07626/) (2018)
- [Eigenoption Discovery through the Deep Successor Representation](https://mlsanity.com/paper/1710.11089/) (2017)
- [Does Zero-Shot Reinforcement Learning Exist?](https://mlsanity.com/paper/2209.14935/) (2022)
- [Proto-Value Networks: Scaling Representation Learning with Auxiliary Tasks](https://mlsanity.com/paper/2304.12567/) (2023)
- [PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning](https://mlsanity.com/paper/2102.12560/) (2021)
- [Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer](https://mlsanity.com/paper/2206.11326/) (2022)
- [Universal Successor Features for Transfer Reinforcement Learning](https://mlsanity.com/paper/2001.04025/) (2020)
- [Prediction and Control in Continual Reinforcement Learning](https://mlsanity.com/paper/2312.11669/) (2023)
- [Combining Behaviors with the Successor Features Keyboard](https://mlsanity.com/paper/2310.15940/) (2023)
- [Advantages and Limitations of using Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1708.00102/) (2017)
- [Composing Task Knowledge with Modular Successor Feature Approximators](https://mlsanity.com/paper/2301.12305/) (2023)
- [Successor Features Combine Elements of Model-Free and Model-based Reinforcement Learning](https://mlsanity.com/paper/1901.11437/) (2019)
- [Successor Feature Representations](https://mlsanity.com/paper/2110.15701/) (2021)

## Built on by (most cited first, 10 of 10)

- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [Reward-Aware Proto-Representations in Reinforcement Learning](https://mlsanity.com/paper/2505.16217/) (2025)
- [Constructing an Optimal Behavior Basis for the Option Keyboard](https://mlsanity.com/paper/2505.00787/) (2025)
- [A Reward-Free Viewpoint on Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2604.24532/) (2026)
- [MAGIK: Mapping to Analogous Goals via Imagination-enabled Knowledge Transfer](https://mlsanity.com/paper/2506.01623/) (2025)
- [Reward-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2603.05066/) (2026)
- [Robust Remote Reinforcement Learning over Unreliable Communication Channels using Homomorphic State Encoding](https://mlsanity.com/paper/2508.07722/) (2025)
- [Squeezing More from the Stream : Learning Representation Online for Streaming Reinforcement Learning](https://mlsanity.com/paper/2602.09396/) (2026)
- [ASPECT:Analogical Semantic Policy Execution via Language Conditioned Transfer](https://mlsanity.com/paper/2604.08355/) (2026)
- [Robust Successor Features](https://mlsanity.com/paper/2609.31016/) (2026)

## Abstract (arXiv)

In Deep Reinforcement Learning (RL), it is a challenge to learn representations that do not exhibit catastrophic forgetting or interference in non-stationary environments. Successor Features (SFs) offer a potential solution to this challenge. However, canonical techniques for learning SFs from pixel-level observations often lead to representation collapse, wherein representations degenerate and fail to capture meaningful variations in the data. More recent methods for learning SFs can avoid representation collapse, but they often involve complex losses and multiple learning phases, reducing their efficiency. We introduce a novel, simple method for learning SFs directly from pixels. Our approach uses a combination of a Temporal-difference (TD) loss and a reward prediction loss, which together capture the basic mathematical definition of SFs. We show that our approach matches or outperforms existing SF learning techniques in both 2D (Minigrid), 3D (Miniworld) mazes and Mujoco, for both single and continual learning scenarios. As well, our technique is efficient, and can reach higher levels of performance in less time than other approaches. Our work provides a new, streamlined technique for learning SFs directly from pixel observations, with no pretraining required.