mlsanityData to 14 Oct 2026

Understanding the theoretical properties of projected Bellman equation, linear Q-learning, and approximate value iteration

arXiv 2504.10865 ↗First posted 15 Apr 2025cs.AI cs.LGcited by 6

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In this paper, we study the theoretical properties of the projected Bellman equation (PBE) and two algorithms to solve this equation: linear Q-learning and approximate value iteration (AVI). We consider two sufficient conditions for the existence of a solution to PBE : strictly negatively row dominating diagonal (SNRDD) assumption and a condition motivated by the convergence of AVI. The SNRDD assumption also ensures the convergence of linear Q-learning, and its relationship with the convergence of AVI is examined. Lastly, several interesting observations on the solution of PBE are provided when using ε-greedy policy.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 50 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

15 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025A Unifying View of Linear Function Approximation in Off-Policy RL Through Matrix Splitting and Preconditioning3 Jan
  2. 2024The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise15 Jan

Other references · 13

  1. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  2. 2018Generalization and Regularization in DQNcited by 118
  3. 2019Finite-Sample Analysis of Nonlinear Stochastic Approximation with Applications in Reinforcement Learningcited by 67
  4. 2022Target Network and Truncation Overcome The Deadly Triad in Q-Learningcited by 31
  5. 2022Regularized Q-learningcited by 16
  6. 2023TD Convergence: An Optimization Perspectivecited by 16
  7. 2023Why Target Networks Stabilise Temporal Difference Methodscited by 13
  8. 2020Periodic Q-Learningcited by 10
  9. 2024Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximationcited by 10
  10. 2022Final Iteration Convergence Bound of Q-Learning: Switching System Approachcited by 9
  11. 2020Understanding the Pathologies of Approximate Policy Evaluation when Combined with Greedification in Reinforcement Learningcited by 7
  12. 2022Convex Programs and Lyapunov Functions for Reinforcement Learning: A Unified Perspective on the Analysis of Value-Based Methodscited by 3
1 more
  1. 2023Non-Euclidean Monotone Operator Theory and Applicationscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026A Switching System Theory of Q-Learning with Linear Function Approximationcited by 2
  2. 2026Target Updates May Stabilize Linear Q-Learning: Periodic and Soft Dynamicscited by 2
  3. 2026Periodic Regularized Q-Learningcited by 1
  4. 2026Optimistic Training and Convergence of Q-Learning -- Extended Versioncited by 1
  5. 2026Geometrically Averaged Hard Target Updates for Linear Q-Learningcited by 0
  6. 2026Limiting-Kernel Q(λ): Bridging Short and Long Horizonscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Q-Learning with Stable Infinite-Dimensional Linear Function Approximation
  2. 2024Linear Bellman Completeness Suffices for Efficient Online Reinforcement Learning with Few Actions
  3. 2024The Role of Inherent Bellman Error in Offline Reinforcement Learning with Linear Function Approximation
  4. 2025Is Bellman Equation Enough for Learning Control?
  5. 2025From Set Convergence to Pointwise Convergence: Finite-Time Guarantees for Average-Reward Q-Learning with Adaptive Stepsizes
  6. 2026Markovian Nonconvex ADMM for Reinforcement Learning: Bellman-Resolvent Stability Beyond Smooth Blocks
  7. 2026Deep Q-Learning on H"older Spaces
  8. 2024Computationally Efficient RL under Linear Bellman Completeness for Deterministic Dynamics