# Approximate Kalman Filter Q-Learning for Continuous State-Space MDPs

arXiv: https://arxiv.org/abs/1309.6868 · HTML: https://mlsanity.com/paper/1309.6868/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2013.

## Built on by (most cited first, 2 of 2)

- [Fast Value Tracking for Deep Reinforcement Learning](https://mlsanity.com/paper/2403.13178/) (2024)
- [Optimizing Algorithms for Mobile Health Interventions with Active Querying Optimization](https://mlsanity.com/paper/2512.08950/) (2025)

## Abstract (arXiv)

We seek to learn an effective policy for a Markov Decision Process (MDP) with continuous states via Q-Learning. Given a set of basis functions over state action pairs we search for a corresponding set of linear weights that minimizes the mean Bellman residual. Our algorithm uses a Kalman filter model to estimate those weights and we have developed a simpler approximate Kalman filter model that outperforms the current state of the art projected TD-Learning methods on several standard benchmark problems.