mlsanityData to 7 Oct 2026

A Unified Switching System Perspective and O.D.E. Analysis of Q-Learning Algorithms

arXiv 1912.02270 ↗First posted 4 Dec 2019math.OC cs.LG cs.SYcited by 14

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce a unified framework for analyzing a large family of Q-learning algorithms, based on switching system perspectives and ODE-based stochastic approximation. We show that the nonlinear ODE models associated with these Q-learning algorithms can be formulated as switched linear systems, and analyze their asymptotic stability by leveraging existing switching system theories. Our approach provides the first O.D.E. analysis of the asymptotic convergence of various Q-learning algorithms, including asynchronous Q-learning and averaging Q-learning. We also extend the approach to analyze Q-learning with linear function approximation and derive a new sufficient condition for its convergence.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 5; 2022: 3; 2023: 2; 2024: 3; 2025: 1.

14 papers.

Sort:

  1. 2021A Lyapunov Theory for Finite-Sample Guarantees of Asynchronous Q-Learning and TD-Learning Variantscited by 57
  2. 2021Breaking the Deadly Triad with a Target Networkcited by 47
  3. 2021A Statistical Analysis of Polyak-Ruppert Averaged Q-learningcited by 29
  4. 2022Regularized Q-learningcited by 16
  5. 2025Linear Q-Learning Does Not Diverge in L2: Convergence Rates to a Bounded Setcited by 11
  6. 2021Truncated Emphatic Temporal Difference Methods for Prediction and Controlcited by 10
  7. 2024Almost Sure Convergence Rates and Concentration of Stochastic Approximation and Reinforcement Learning with Markovian Noisecited by 10
  8. 2023Stability of Q-Learning Through Design and Optimismcited by 7
  9. 2021Finite-Sample Analysis of Decentralized Q-Learning for Stochastic Gamescited by 4
  10. 2023Backstepping Temporal Difference Learningcited by 2
  11. 2022Sufficient Exploration for Convex Q-learningcited by 1
  12. 2024Regularized Q-Learning with Linear Function Approximationcited by 1
2 more
  1. 2022Exact Formulas for Finite-Time Estimation Errors of Decentralized Temporal Difference Learning with Linear Function Approximationcited by 0
  2. 2024Learning the Model While Learning Q: Finite-Time Sample Complexity of Online SyncMBQcited by 0