# Probabilistic Successor Representations with Kalman Temporal Differences

arXiv: https://arxiv.org/abs/1910.02532 · HTML: https://mlsanity.com/paper/1910.02532/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2019.

## Built on by (most cited first, 5 of 5)

- [Memory, Space, and Planning: Multiscale Predictive Representations](https://mlsanity.com/paper/2401.09491/) (2024)
- [Multi-Agent Reinforcement Learning via Adaptive Kalman Temporal Difference and Successor Representation](https://mlsanity.com/paper/2112.15156/) (2021)
- [AKF-SR: Adaptive Kalman Filtering-based Successor Representation](https://mlsanity.com/paper/2204.00049/) (2022)
- [Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning](https://mlsanity.com/paper/2310.10818/) (2023)
- [Generalized Kalman filter based temporal difference reinforcement learning](https://mlsanity.com/paper/2607.20010/) (2026)

## Abstract (arXiv)

The effectiveness of Reinforcement Learning (RL) depends on an animal's ability to assign credit for rewards to the appropriate preceding stimuli. One aspect of understanding the neural underpinnings of this process involves understanding what sorts of stimulus representations support generalisation. The Successor Representation (SR), which enforces generalisation over states that predict similar outcomes, has become an increasingly popular model in this space of inquiries. Another dimension of credit assignment involves understanding how animals handle uncertainty about learned associations, using probabilistic methods such as Kalman Temporal Differences (KTD). Combining these approaches, we propose using KTD to estimate a distribution over the SR. KTD-SR captures uncertainty about the estimated SR as well as covariances between different long-term predictions. We show that because of this, KTD-SR exhibits partial transition revaluation as humans do in this experiment without additional replay, unlike the standard TD-SR algorithm. We conclude by discussing future applications of the KTD-SR as a model of the interaction between predictive and probabilistic animal reasoning.