mlsanityData to 7 Oct 2026

Regularized Q-learning

arXiv 2202.05404 ↗First posted 11 Feb 2022cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Q-learning is widely used algorithm in reinforcement learning community. Under the lookup table setting, its convergence is well established. However, its behavior is known to be unstable with the linear function approximation case. This paper develops a new Q-learning algorithm that converges when linear function approximation is used. We prove that simply adding an appropriate regularization term ensures convergence of the algorithm. We prove its stability using a recent analysis tool based on switching system models. Moreover, we experimentally show that it converges in environments where Q-learning with linear function approximation has known to diverge. We also provide an error bound on the solution where the algorithm converges.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 15 of its 58 references that are in the map. First posted on arXiv 11 Feb 2022.

Builds on

15 of 58 references.

References · 15

  1. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  2. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  3. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  4. 2019A Theory of Regularized Markov Decision Processescited by 279
  5. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  6. 2018Generalization and Regularization in DQNcited by 117
  7. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  8. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  9. 2021Breaking the Deadly Triad with a Target Networkcited by 47
  10. 2020Gradient Temporal-Difference Learning with Regularized Correctionscited by 38
  11. 2014Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spacescited by 19
  12. 2021Online Target Q-learning with Reverse Experience Replay: Efficiently finding the Optimal Policy for Linear MDPscited by 19
3 more
  1. 2019A Unified Switching System Perspective and O.D.E. Analysis of Q-Learning Algorithmscited by 14
  2. 2021New Versions of Gradient Temporal Difference Learningcited by 6
  3. 2019A Convergent Off-Policy Temporal Difference Algorithmcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 1; 2025: 5; 2026: 7.

16 papers.

Sort:

  1. 2023TD Convergence: An Optimization Perspectivecited by 15
  2. 2025Linear Q-Learning Does Not Diverge in L2: Convergence Rates to a Bounded Setcited by 11
  3. 2023Stability of Q-Learning Through Design and Optimismcited by 7
  4. 2025Understanding the theoretical properties of projected Bellman equation, linear Q-learning, and approximate value iterationcited by 6
  5. 2025O(1/k) Finite-Time Bound for Non-Linear Two-Time-Scale Stochastic Approximationcited by 5
  6. 2025Extensions of Robbins-Siegmund Theorem with Applications in Reinforcement Learningcited by 5
  7. 2023Multi-Bellman operator for convergence of Q-learning with linear function approximationcited by 3
  8. 2026A Switching System Theory of Q-Learning with Linear Function Approximationcited by 2
  9. 2026Bellman Residual Minimization for Control: Geometry, Stationarity, and Convergencecited by 1
  10. 2026Optimistic Training and Convergence of Q-Learning -- Extended Versioncited by 1
  11. 2024Regularized Q-learning through Robust Averagingcited by 0
  12. 2025Adaptive Policy Backbone via Shared Networkcited by 0
4 more
  1. 2026Periodic Regularized Q-Learningcited by 0
  2. 2026Contraction-Aligned Analysis of Soft Bellman Residual Minimization with Weighted Lp-Norm for Markov Decision Problemcited by 0
  3. 2026Geometrically Averaged Hard Target Updates for Linear Q-Learningcited by 0
  4. 2026Limiting-Kernel Q(λ): Bridging Short and Long Horizonscited by 0