# Switching Theory for Q-Learning

arXiv: https://arxiv.org/abs/2604.19569 · HTML: https://mlsanity.com/paper/2604.19569/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 1 of its 32 references that are in the map.
- First posted on arXiv 21 Apr 2026.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 1 of 1)

- [Stochastic approximation with cone-contractive operators: Sharp ℓ_∞-bounds for Q-learning](https://mlsanity.com/paper/1905.06265/) (2019)

## Built on by (most cited first, 7 of 7)

- [A Switching System Theory of Q-Learning with Linear Function Approximation](https://mlsanity.com/paper/2605.11021/) (2026)
- [Target Updates May Stabilize Linear Q-Learning: Periodic and Soft Dynamics](https://mlsanity.com/paper/2606.02645/) (2026)
- [Switching-Geometry Analysis of Deflated Q-Value Iteration](https://mlsanity.com/paper/2605.10811/) (2026)
- [Sign-Separated Asymmetric Finite-Time Error Analysis of Q-Learning](https://mlsanity.com/paper/2605.16103/) (2026)
- [Geometrically Averaged Hard Target Updates for Linear Q-Learning](https://mlsanity.com/paper/2606.10835/) (2026)
- [Heavy-Ball Q-Learning with Residual Weighting Correction](https://mlsanity.com/paper/2606.27112/) (2026)
- [Spectral Analysis of Dueling Q-Learning](https://mlsanity.com/paper/2607.08340/) (2026)

## Abstract (arXiv)

Q-learning is a fundamental algorithmic primitive in reinforcement learning. This paper develops a new framework for analyzing constant step-size tabular Q-learning from a switching linear system (SLS) viewpoint. In particular, we derive a stochastic SLS representation of the Q-learning error, and a finite-time error analysis through the joint spectral radius (JSR) of the corresponding SLS model, where the JSR is the exact worst-case exponential rate of the associated SLS. To the best of our knowledge, this is the first convergence rate analysis of standard Q-learning whose leading exponential rate is expressed through the JSR. The resulting rate is tied to the intrinsic worst-case exponential rate of the direct SLS representation and can be sharper than row-sum upper bounds when those bounds are conservative. We further prove that the JSR of Q-learning equals the largest spectral radius among the deterministic-policy modes and give an exact linear programming characterization that can be evaluated to any prescribed accuracy.