mlsanityData to 7 Oct 2026

VA-learning as a more efficient alternative to Q-learning

arXiv 2305.18161 ↗First posted 29 May 2023cs.LG stat.MLcited by 11

Not on the map. Why?

Abstract (arXiv)

In reinforcement learning, the advantage function is critical for policy improvement, but is often extracted from a learned Q-function. A natural question is: Why not learn the advantage function directly? In this work, we introduce VA-learning, which directly learns advantage function and value function using bootstrapping, without explicit reference to Q-functions. VA-learning learns off-policy and enjoys similar theoretical guarantees as Q-learning. Thanks to the direct learning of advantage function and value function, VA-learning improves the sample efficiency over Q-learning both in tabular implementations and deep RL agents on Atari-57 games. We also identify a close connection between VA-learning and the dueling architecture, which partially explains why a simple architectural change to DQN agents tends to improve performance.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 9 of its 19 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

9 of 19 references.

References · 9

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  4. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  5. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  6. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
  7. 2015Increasing the Action Gap: New Operators for Reinforcement Learningcited by 49
  8. 2021Direct Advantage Estimationcited by 11
  9. 2019Gap-Increasing Policy Evaluation for Efficient and Noise-Tolerant Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 3; 2026: 4.

11 papers.

  1. 2023Generative Flow Networks as Entropy-Regularized RLcited by 60
  2. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  3. 2025Soft Policy Optimization: Online Off-Policy RL for Sequence Modelscited by 12
  4. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learningcited by 7
  5. 2024On the Low-Rank Parametrization of Reward Models for Controlled Language Generationcited by 3
  6. 2025An Analysis of Action-Value Temporal-Difference Methods That Learn State Valuescited by 2
  7. 2025Multi-Agent Reinforcement Learning for Greenhouse Gas Offset Credit Marketscited by 1
  8. 2026Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learningcited by 1
  9. 2026Accelerating Q-learning through Efficient Value-Sharing across Actionscited by 0
  10. 2026Spectral Analysis of Dueling Q-Learningcited by 0
  11. 2026Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learningcited by 0