# Predecessor Features

arXiv: https://arxiv.org/abs/2206.00303 · HTML: https://mlsanity.com/paper/2206.00303/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 6 references that are in the map.
- First posted on arXiv 1 Jun 2022.

## Builds on (most cited first, 4 of 4)

- [Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvement](https://mlsanity.com/paper/1901.10964/) (2019)
- [Expected Eligibility Traces](https://mlsanity.com/paper/2007.01839/) (2020)
- [Successor Features Combine Elements of Model-Free and Model-based Reinforcement Learning](https://mlsanity.com/paper/1901.11437/) (2019)
- [Source Traces for Temporal Difference Learning](https://mlsanity.com/paper/1902.02907/) (2019)

## Built on by (most cited first, 2 of 2)

- [Maximum State Entropy Exploration using Predecessor and Successor Representations](https://mlsanity.com/paper/2306.14808/) (2023)
- [Successor-Predecessor Intrinsic Exploration](https://mlsanity.com/paper/2305.15277/) (2023)

## Abstract (arXiv)

Any reinforcement learning system must be able to identify which past events contributed to observed outcomes, a problem known as credit assignment. A common solution to this problem is to use an eligibility trace to assign credit to recency-weighted set of experienced events. However, in many realistic tasks, the set of recently experienced events are only one of the many possible action events that could have preceded the current outcome. This suggests that reinforcement learning can be made more efficient by allowing credit assignment to any viable preceding state, rather than only those most recently experienced. Accordingly, we examine ``Predecessor Features'', the fully bootstrapped version of van Hasselt's ``Expected Trace'', an algorithm that achieves this richer form of credit assignment. By maintaining a representation that approximates the expected sum of past occupancies, this algorithm allows temporal difference (TD) errors to be propagated accurately to a larger number of predecessor states than conventional methods, greatly improving learning speed. The algorithm can also be naturally extended from tabular state representation to feature representations allowing for increased performance on a wide range of environments. We demonstrate several use cases for Predecessor Features and compare its performance with other approaches.