# Target Updates May Stabilize Linear Q-Learning: Periodic and Soft Dynamics

arXiv: https://arxiv.org/abs/2606.02645 · HTML: https://mlsanity.com/paper/2606.02645/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 11 of its 30 references that are in the map.
- First posted on arXiv 31 May 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 11 of 11)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Breaking the Deadly Triad with a Target Network](https://mlsanity.com/paper/2101.08862/) (2021)
- [Target Network and Truncation Overcome The Deadly Triad in Q-Learning](https://mlsanity.com/paper/2203.02628/) (2022)
- [Target-Based Temporal Difference Learning](https://mlsanity.com/paper/1904.10945/) (2019)
- [A Discrete-Time Switching System Analysis of Q-learning](https://mlsanity.com/paper/2102.08583/) (2021)
- [Why Target Networks Stabilise Temporal Difference Methods](https://mlsanity.com/paper/2302.12537/) (2023)
- [Periodic Q-Learning](https://mlsanity.com/paper/2002.09795/) (2020)
- [Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation](https://mlsanity.com/paper/2405.21043/) (2024)
- [Switching Theory for Q-Learning](https://mlsanity.com/paper/2604.19569/) (2026)
- [Understanding the theoretical properties of projected Bellman equation, linear Q-learning, and approximate value iteration](https://mlsanity.com/paper/2504.10865/) (2025)
- [A Switching System Theory of Q-Learning with Linear Function Approximation](https://mlsanity.com/paper/2605.11021/) (2026)

## Built on by (most cited first, 2 of 2)

- [Geometrically Averaged Hard Target Updates for Linear Q-Learning](https://mlsanity.com/paper/2606.10835/) (2026)
- [A Contraction Framework for Stochastic Operators with Bootstrapping: Application to TD Learning](https://mlsanity.com/paper/2609.29961/) (2026)

## Abstract (arXiv)

Periodic target updates in Q-learning and soft target updates in actor-critic methods are empirically well established stabilization mechanisms, but their precise theoretical explanation is still incomplete. This paper gives a rigorous and exact analysis of these mechanisms for Q-learning with linear function approximation (linear Q-learning) using the exact switched linear system (SLS) dynamics induced by the Bellman maximum and the joint spectral radius (JSR) of the resulting switching matrix families. Although linear Q-learning can fail to converge in general, we prove that, under explicit spectral and step-size conditions, periodic hard target updates and soft target updates can guarantee convergence to the exact projected Q-Bellman solution. The main analysis is carried out for deterministic linear Q-learning, where the target-update mechanism is most transparent. Once the corresponding JSR certificate is established for the mean recursion, the stochastic reinforcement-learning setting can be treated by replacing deterministic modes with sampled stochastic modes and adding the corresponding stochastic-noise analysis.