# Heavy-Ball Q-Learning with Residual Weighting Correction

arXiv: https://arxiv.org/abs/2606.27112 · HTML: https://mlsanity.com/paper/2606.27112/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 8 of its 29 references that are in the map.
- First posted on arXiv 25 Jun 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 8 of 8)

- [Analysis of Q-learning with Adaptation and Momentum Restart for Gradient Descent](https://mlsanity.com/paper/2007.07422/) (2020)
- [A First-Order Approach To Accelerated Value Iteration](https://mlsanity.com/paper/1905.09963/) (2019)
- [Anderson Acceleration for Reinforcement Learning](https://mlsanity.com/paper/1809.09501/) (2018)
- [Accelerating Value Iteration with Anchoring](https://mlsanity.com/paper/2305.16569/) (2023)
- [Switching Theory for Q-Learning](https://mlsanity.com/paper/2604.19569/) (2026)
- [Deflated Dynamics Value Iteration](https://mlsanity.com/paper/2407.10454/) (2024)
- [Switching-Geometry Analysis of Deflated Q-Value Iteration](https://mlsanity.com/paper/2605.10811/) (2026)
- [Spectral Analysis of Heavy-Ball Q-value Iteration](https://mlsanity.com/paper/2607.22627/) (2026)

## Abstract (arXiv)

This paper proposes a corrected heavy-ball Q-learning method for reinforcement learning (RL) and establishes convergence of its deterministic mean dynamics. It also identifies conditions under which the method is theoretically guaranteed to converge faster than standard Q-learning. The same construction is then extended to Q-learning with linear function approximation, where analogous convergence and acceleration statements are derived for the corresponding corrected fixed point. The sampled stochastic versions are treated through conditional-mean recursions and, in the stated linear-function-approximation setting, finite-time bounds. The analysis is based on a switched linear system (SLS) representation of Q-learning algorithms and on the joint spectral radius (JSR) of the associated switching families. This SLS viewpoint is not commonly used in standard analyses of Q-learning, and it provides a complementary framework and new insight into how heavy-ball momentum can accelerate Q-learning.