# VA-learning as a more efficient alternative to Q-learning

arXiv: https://arxiv.org/abs/2305.18161 · HTML: https://mlsanity.com/paper/2305.18161/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 19 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 9 of 9)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Sample Efficient Actor-Critic with Experience Replay](https://mlsanity.com/paper/1611.01224/) (2016)
- [Increasing the Action Gap: New Operators for Reinforcement Learning](https://mlsanity.com/paper/1512.04860/) (2015)
- [Direct Advantage Estimation](https://mlsanity.com/paper/2109.06093/) (2021)
- [Gap-Increasing Policy Evaluation for Efficient and Noise-Tolerant Reinforcement Learning](https://mlsanity.com/paper/1906.07586/) (2019)

## Built on by (most cited first, 11 of 11)

- [Generative Flow Networks as Entropy-Regularized RL](https://mlsanity.com/paper/2310.12934/) (2023)
- [Value Augmented Sampling for Language Model Alignment and Personalization](https://mlsanity.com/paper/2405.06639/) (2024)
- [Soft Policy Optimization: Online Off-Policy RL for Sequence Models](https://mlsanity.com/paper/2503.05453/) (2025)
- [Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning](https://mlsanity.com/paper/2410.11022/) (2024)
- [On the Low-Rank Parametrization of Reward Models for Controlled Language Generation](https://mlsanity.com/paper/2407.04615/) (2024)
- [An Analysis of Action-Value Temporal-Difference Methods That Learn State Values](https://mlsanity.com/paper/2507.09523/) (2025)
- [Multi-Agent Reinforcement Learning for Greenhouse Gas Offset Credit Markets](https://mlsanity.com/paper/2504.11258/) (2025)
- [Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2606.20411/) (2026)
- [Accelerating Q-learning through Efficient Value-Sharing across Actions](https://mlsanity.com/paper/2606.29806/) (2026)
- [Spectral Analysis of Dueling Q-Learning](https://mlsanity.com/paper/2607.08340/) (2026)
- [Endpoint Replay: Compressing the Recency Buffer in Deep Reinforcement Learning](https://mlsanity.com/paper/2607.25123/) (2026)

## Abstract (arXiv)

In reinforcement learning, the advantage function is critical for policy improvement, but is often extracted from a learned Q-function. A natural question is: Why not learn the advantage function directly? In this work, we introduce VA-learning, which directly learns advantage function and value function using bootstrapping, without explicit reference to Q-functions. VA-learning learns off-policy and enjoys similar theoretical guarantees as Q-learning. Thanks to the direct learning of advantage function and value function, VA-learning improves the sample efficiency over Q-learning both in tabular implementations and deep RL agents on Atari-57 games. We also identify a close connection between VA-learning and the dueling architecture, which partially explains why a simple architectural change to DQN agents tends to improve performance.