# Trust Region Value Optimization using Kalman Filtering

arXiv: https://arxiv.org/abs/1901.07860 · HTML: https://mlsanity.com/paper/1901.07860/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2019.

## Built on by (most cited first, 6 of 6)

- [Bridging the Gap Between Target Networks and Functional Regularization](https://mlsanity.com/paper/2106.02613/) (2021)
- [KOALA: A Kalman Optimization Algorithm with Loss Adaptivity](https://mlsanity.com/paper/2107.03331/) (2021)
- [KOALA++: Efficient Kalman-Based Optimization with Gradient-Covariance Products](https://mlsanity.com/paper/2506.04432/) (2025)
- [LoKO: Low-Rank Kalman Optimizer for Online Fine-Tuning of Large Models](https://mlsanity.com/paper/2410.11551/) (2024)
- [Bridging the Gap Between Target Networks and Functional Regularization](https://mlsanity.com/paper/2210.12282/) (2022)
- [K-Score: Kalman Filter as a Principled Alternative to Reward Normalization in Reinforcement Learning](https://mlsanity.com/paper/2604.23056/) (2026)

## Abstract (arXiv)

Policy evaluation is a key process in reinforcement learning. It assesses a given policy using estimation of the corresponding value function. When using a parameterized function to approximate the value, it is common to optimize the set of parameters by minimizing the sum of squared Bellman Temporal Differences errors. However, this approach ignores certain distributional properties of both the errors and value parameters. Taking these distributions into account in the optimization process can provide useful information on the amount of confidence in value estimation. In this work we propose to optimize the value by minimizing a regularized objective function which forms a trust region over its parameters. We present a novel optimization method, the Kalman Optimization for Value Approximation (KOVA), based on the Extended Kalman Filter. KOVA minimizes the regularized objective function by adopting a Bayesian perspective over both the value parameters and noisy observed returns. This distributional property provides information on parameter uncertainty in addition to value estimates. We provide theoretical results of our approach and analyze the performance of our proposed optimizer on domains with large state and action spaces.