# Generalized Kalman filter based temporal difference reinforcement learning

arXiv: https://arxiv.org/abs/2607.20010 · HTML: https://mlsanity.com/paper/2607.20010/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 13 of its 50 references that are in the map.
- First posted on arXiv 22 Jul 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 13 of 13)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Controlling Rayleigh-B'enard convection via Reinforcement Learning](https://mlsanity.com/paper/2003.14358/) (2020)
- [Generative Design by Reinforcement Learning: Enhancing the Diversity of Topology Optimization Designs](https://mlsanity.com/paper/2008.07119/) (2020)
- [Kalman Temporal Differences](https://mlsanity.com/paper/1406.3270/) (2014)
- [MM-KTD: Multiple Model Kalman Temporal Differences for Reinforcement Learning](https://mlsanity.com/paper/2006.00195/) (2020)
- [Adaptive Stochastic Nonlinear Model Predictive Control with Look-ahead Deep Reinforcement Learning for Autonomous Vehicle Motion Control](https://mlsanity.com/paper/2311.04303/) (2023)
- [SPD Matrix Learning for Neuroimaging Analysis: Perspectives, Methods, and Challenges](https://mlsanity.com/paper/2504.18882/) (2025)
- [Probabilistic Successor Representations with Kalman Temporal Differences](https://mlsanity.com/paper/1910.02532/) (2019)
- [Robust and Adaptive Temporal-Difference Learning Using An Ensemble of Gaussian Processes](https://mlsanity.com/paper/2112.00882/) (2021)
- [Semiparametric forecasting and filtering: correcting low-dimensional model error in parametric models](https://mlsanity.com/paper/1502.07766/) (2015)
- [Analytically Tractable Bayesian Deep Q-Learning](https://mlsanity.com/paper/2106.11086/) (2021)

## Abstract (arXiv)

In this paper, we present a generalized temporal-difference (TD) reinforcement learning framework based on the theory of conditional expectations. The value and action-value (Q-value) functions are treated as uncertain quantities, and their estimation is formulated as a stochastic inference problem. Unlike classical Kalman-based temporal-difference learning, which relies on linear-Gaussian assumptions, the proposed formulation is derived directly from the conditional expectation framework and naturally extends to nonlinear models and non-Gaussian probability distributions. The proposed method recursively estimates not only the conditional expectation of the value function but also its second probabilistic moment, thereby quantifying the uncertainty associated with the learned value function throughout the learning process. To obtain a computationally tractable algorithm, the stochastic problem is discretized using either polynomial chaos expansions or ensemble-based approximations, providing efficient representations of the underlying random variables. The proposed framework is demonstrated on two optimal control problems: a linear mass--spring--damper system and a nonlinear heat conduction problem in a closed cavity. The numerical examples illustrate the capability of the proposed method to accurately estimate both the value function and its associated uncertainty, while extending classical Kalman-based temporal-difference learning to a broader class of stochastic systems.