# AKF-SR: Adaptive Kalman Filtering-based Successor Representation

arXiv: https://arxiv.org/abs/2204.00049 · HTML: https://mlsanity.com/paper/2204.00049/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 58 references that are in the map.
- First posted on arXiv 31 Mar 2022.

## Builds on (most cited first, 15 of 15)

- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [A General and Adaptive Robust Loss Function](https://mlsanity.com/paper/1701.03077/) (2017)
- [Model-Based Reinforcement Learning with Value-Targeted Regression](https://mlsanity.com/paper/2006.01107/) (2020)
- [Count-Based Exploration with the Successor Representation](https://mlsanity.com/paper/1807.11622/) (2018)
- [Deep Successor Reinforcement Learning](https://mlsanity.com/paper/1606.02396/) (2016)
- [Measuring the Reliability of Reinforcement Learning Algorithms](https://mlsanity.com/paper/1912.05663/) (2019)
- [Adaptive Adjustment of Noise Covariance in Kalman Filter for Dynamic State Estimation](https://mlsanity.com/paper/1702.00884/) (2017)
- [A neurally plausible model learns successor representations in partially observable environments](https://mlsanity.com/paper/1906.09480/) (2019)
- [A Complementary Learning Systems Approach to Temporal Difference Learning](https://mlsanity.com/paper/1905.02636/) (2019)
- [Kalman Temporal Differences](https://mlsanity.com/paper/1406.3270/) (2014)
- [Universal Successor Representations for Transfer Reinforcement Learning](https://mlsanity.com/paper/1804.03758/) (2018)
- [Kalman meets Bellman: Improving Policy Evaluation through Value Tracking](https://mlsanity.com/paper/2002.07171/) (2020)
- [MM-KTD: Multiple Model Kalman Temporal Differences for Reinforcement Learning](https://mlsanity.com/paper/2006.00195/) (2020)
- [Recovering Robustness in Model-Free Reinforcement learning](https://mlsanity.com/paper/1810.09337/) (2018)
- [Probabilistic Successor Representations with Kalman Temporal Differences](https://mlsanity.com/paper/1910.02532/) (2019)

## Built on by (most cited first, 1 of 1)

- [Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning](https://mlsanity.com/paper/2310.10818/) (2023)

## Abstract (arXiv)

Recent studies in neuroscience suggest that Successor Representation (SR)-based models provide adaptation to changes in the goal locations or reward function faster than model-free algorithms, together with lower computational cost compared to that of model-based algorithms. However, it is not known how such representation might help animals to manage uncertainty in their decision-making. Existing methods for SR learning do not capture uncertainty about the estimated SR. In order to address this issue, the paper presents a Kalman filter-based SR framework, referred to as Adaptive Kalman Filtering-based Successor Representation (AKF-SR). First, Kalman temporal difference approach, which is a combination of the Kalman filter and the temporal difference method, is used within the AKF-SR framework to cast the SR learning procedure into a filtering problem to benefit from the uncertainty estimation of the SR, and also decreases in memory requirement and sensitivity to model's parameters in comparison to deep neural network-based algorithms. An adaptive Kalman filtering approach is then applied within the proposed AKF-SR framework in order to tune the measurement noise covariance and measurement mapping function of Kalman filter as the most important parameters affecting the filter's performance. Moreover, an active learning method that exploits the estimated uncertainty of the SR to form the behaviour policy leading to more visits to less certain values is proposed to improve the overall performance of an agent in terms of received rewards while interacting with its environment.