Generalized Kalman filter based temporal difference reinforcement learning
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
In this paper, we present a generalized temporal-difference (TD) reinforcement learning framework based on the theory of conditional expectations. The value and action-value (Q-value) functions are treated as uncertain quantities, and their estimation is formulated as a stochastic inference problem. Unlike classical Kalman-based temporal-difference learning, which relies on linear-Gaussian assumptions, the proposed formulation is derived directly from the conditional expectation framework and naturally extends to nonlinear models and non-Gaussian probability distributions. The proposed method recursively estimates not only the conditional expectation of the value function but also its second probabilistic moment, thereby quantifying the uncertainty associated with the learned value function throughout the learning process. To obtain a computationally tractable algorithm, the stochastic problem is discretized using either polynomial chaos expansions or ensemble-based approximations, providing efficient representations of the underlying random variables. The proposed framework is demonstrated on two optimal control problems: a linear mass--spring--damper system and a nonlinear heat conduction problem in a closed cavity. The numerical examples illustrate the capability of the proposed method to accurately estimate both the value function and its associated uncertainty, while extending classical Kalman-based temporal-difference learning to a broader class of stochastic systems.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 50 references that are in the map. First posted on arXiv 22 Jul 2026.
Builds on
13 of 50 references.
References · 13
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
- 2020Controlling Rayleigh-B'enard convection via Reinforcement Learningcited by 25
- 2020Generative Design by Reinforcement Learning: Enhancing the Diversity of Topology Optimization Designscited by 19
- 2014Kalman Temporal Differencescited by 10
- 2020MM-KTD: Multiple Model Kalman Temporal Differences for Reinforcement Learningcited by 6
- 2023Adaptive Stochastic Nonlinear Model Predictive Control with Look-ahead Deep Reinforcement Learning for Autonomous Vehicle Motion Controlcited by 6
- 2025SPD Matrix Learning for Neuroimaging Analysis: Perspectives, Methods, and Challengescited by 6
- 2019Probabilistic Successor Representations with Kalman Temporal Differencescited by 5
- 2021Robust and Adaptive Temporal-Difference Learning Using An Ensemble of Gaussian Processescited by 4
- 2015Semiparametric forecasting and filtering: correcting low-dimensional model error in parametric modelscited by 2
1 more
- 2021Analytically Tractable Bayesian Deep Q-Learningcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Towards Parameter-Free Temporal Difference Learning
- 2024An MRP Formulation for Supervised Learning: Generalized Temporal Difference Learning Models
- 2024Statistical Inference for Policy Evaluation with Temporal Difference Learning
- 2024Dual Ensemble Kalman Filter for Stochastic Optimal Control ↗
- 2025A Temporal Difference Method for Stochastic Continuous Dynamics
- 2025Stabilizing Temporal Difference Learning via Implicit Stochastic Recursion
- 2024Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features
- 2025Uncertainty quantification for Markov chain induced martingales with application to temporal difference learning