A Variance Minimization Approach to Temporal-Difference Learning
On the map: Deep reinforcement learning methods → RL theory and sample complexity
Abstract (arXiv)
Fast-converging algorithms are a contemporary requirement in reinforcement learning. In the context of linear function approximation, the magnitude of the smallest eigenvalue of the key matrix is a major factor reflecting the convergence speed. Traditional value-based RL algorithms focus on minimizing errors. This paper introduces a variance minimization (VM) approach for value-based RL instead of error minimization. Based on this approach, we proposed two objectives, the Variance of Bellman Error (VBE) and the Variance of Projected Bellman Error (VPBE), and derived the VMTD, VMTDC, and VMETD algorithms. We provided proofs of their convergence and optimal policy invariance of the variance minimization. Experimental studies validate the effectiveness of the proposed algorithms.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 0 of its 29 references that are in the map. First posted on arXiv 10 Nov 2024.
Builds on
0 of 29 references.
No references here yet.
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Statistical Inference for Policy Evaluation with Temporal Difference Learning
- 2026Mathematical methods of reinforcement learning
- 2026Towards Parameter-Free Temporal Difference Learning
- 2025Stabilizing Temporal Difference Learning via Implicit Stochastic Recursion
- 2024PID Accelerated Temporal Difference Algorithms
- 2025Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards
- 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Space
- 2026V0.5: Generalist Value Model as a Prior for Sparse RL Rollouts