# Which Features are Best for Successor Features?

arXiv: https://arxiv.org/abs/2502.10790 · HTML: https://mlsanity.com/paper/2502.10790/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 19 of its 22 references that are in the map.
- First posted on arXiv 15 Feb 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 19 of 19)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [Fast Task Inference with Variational Intrinsic Successor Features](https://mlsanity.com/paper/1906.05030/) (2019)
- [Learning One Representation to Optimize All Rewards](https://mlsanity.com/paper/2103.07945/) (2021)
- [APS: Active Pretraining with Successor Features](https://mlsanity.com/paper/2108.13956/) (2021)
- [Universal Successor Features Approximators](https://mlsanity.com/paper/1812.07626/) (2018)
- [The Laplacian in RL: Learning Representations with Efficient Approximations](https://mlsanity.com/paper/1810.04586/) (2018)
- [Does Zero-Shot Reinforcement Learning Exist?](https://mlsanity.com/paper/2209.14935/) (2022)
- [Learning Successor States and Goal-Dependent Values: A Mathematical Viewpoint](https://mlsanity.com/paper/2101.07123/) (2021)
- [Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments](https://mlsanity.com/paper/1612.05533/) (2016)
- [Spectral Decomposition Representation for Reinforcement Learning](https://mlsanity.com/paper/2208.09515/) (2022)
- [A New Representation of Successor Features for Transfer across Dissimilar Environments](https://mlsanity.com/paper/2107.08426/) (2021)
- [Disentangled Cumulants Help Successor Representations Transfer to New Tasks](https://mlsanity.com/paper/1911.10866/) (2019)
- [Approximate Temporal Difference Learning is a Gradient Descent for Reversible Policies](https://mlsanity.com/paper/1805.00869/) (2018)
- [Unbiased Methods for Multi-Goal Reinforcement Learning](https://mlsanity.com/paper/2106.08863/) (2021)

## Built on by (most cited first, 3 of 3)

- [The Laplacian Keyboard: Beyond the Linear Span](https://mlsanity.com/paper/2602.07730/) (2026)
- [The Geometry of Empowerment](https://mlsanity.com/paper/2610.07796/) (2026)
- [Task Inference Beyond Least Squares in Behavioral Foundation Models](https://mlsanity.com/paper/2610.05350/) (2026)

## Abstract (arXiv)

In reinforcement learning, universal successor features (SFs) are a way to provide zero-shot adaptation to new tasks at test time: they provide optimal policies for all downstream reward functions lying in the linear span of a set of base features. But it is unclear what constitutes a good set of base features, that could be useful for a wide set of downstream tasks beyond their linear span. Laplacian eigenfunctions (the eigenfunctions of Delta+Delta^ast with Delta the Laplacian operator of some reference policy and Delta^ast that of the time-reversed dynamics) have been argued to play a role, and offer good empirical performance. Here, for the first time, we identify the optimal base features based on an objective criterion of downstream performance, in a non-tautological way without assuming the downstream tasks are linear in the features. We do this for three generic classes of downstream tasks: reaching a random goal state, dense random Gaussian rewards, and random ``scattered'' sparse rewards. The features yielding optimal expected downstream performance turn out to be the same for these three task families. They do not coincide with Laplacian eigenfunctions in general, though they can be expressed from Delta: in the simplest case (deterministic environment and decay factor γ close to 1), they are the eigenfunctions of Delta-1+(Delta-1)^ast. We obtain these results under an assumption of large behavior cloning regularization with respect to a reference policy, a setting often used for offline RL. Along the way, we get new insights into KL-regularizedoptionnatural policy gradient, and into the lack of SF information in the norm of Bellman gaps.