mlsanityData to 14 Oct 2026

Multi-Bellman operator for convergence of Q-learning with linear function approximation

arXiv 2309.16819 ↗First posted 28 Sep 2023cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

We study the convergence of Q-learning with linear function approximation. Our key contribution is the introduction of a novel multi-Bellman operator that extends the traditional Bellman operator. By exploring the properties of this operator, we identify conditions under which the projected multi-Bellman operator becomes contractive, providing improved fixed-point guarantees compared to the Bellman operator. To leverage these insights, we propose the multi Q-learning algorithm with linear function approximation. We demonstrate that this algorithm converges to the fixed-point of the projected multi-Bellman operator, yielding solutions of arbitrary accuracy. Finally, we validate our approach by applying it to well-known environments, showcasing the effectiveness and applicability of our findings.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 16 of its 53 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

16 of 53 references.

References · 16

  1. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  2. 2017Stochastic Variance Reduction Methods for Policy Evaluationcited by 58
  3. 2019A Finite-Time Analysis of Q-Learning with Neural Network Function Approximationcited by 47
  4. 2021Breaking the Deadly Triad with a Target Networkcited by 47
  5. 2017Multi-step Reinforcement Learning: A Unifying Algorithmcited by 42
  6. 2022Target Network and Truncation Overcome The Deadly Triad in Q-Learningcited by 31
  7. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximationcited by 22
  8. 2018Beyond the One Step Greedy Approach in Reinforcement Learningcited by 20
  9. 2014Proximal Reinforcement Learning: A New Theory of Sequential Decision Making in Primal-Dual Spacescited by 19
  10. 2021Online Target Q-learning with Reverse Experience Replay: Efficiently finding the Optimal Policy for Linear MDPscited by 19
  11. 2022Regularized Q-learningcited by 16
  12. 2021Convergent and Efficient Deep Q Network Algorithmcited by 8
4 more
  1. 2022Planning and Learning with Adaptive Lookaheadcited by 8
  2. 2013Estimating the Maximum Expected Value: An Analysis of (Nested) Cross Validation and the Maximum Sample Averagecited by 7
  3. 2020Think Too Fast Nor Too Slow: The Computational Trade-off Between Planning And Reinforcement Learningcited by 5
  4. 2022Reinforcement Learning with Unbiased Policy Evaluation and Linear Function Approximationcited by 4

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Beyond expected value: geometric mean optimization for long-term policy performance in reinforcement learningcited by 2
  2. 2025Analysis of Off-Policy n-Step TD-Learning with Linear Function Approximationcited by 0
  3. 2025Networked Restless Multi-Arm Bandits with Reinforcement Learningcited by 0