mlsanityData to 14 Oct 2026

Relative Value Learning

arXiv 2607.21120 ↗First posted 23 Jul 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In reinforcement learning, critics typically estimate absolute state values V(s), estimating how good a particular situation is in isolation. However, it turns out that only differences in value are relevant for control. Motivated by this, we propose Relative Value Learning (RV), a framework that learns value differences directly via an antisymmetric function Delta(si, sj) = V(si) - V(sj). We introduce a pairwise Bellman operator and prove it is a γ-contraction with a unique fixed point equal to the true value differences, derive well-posed 1-step, n-step and λ-return targets and reconstruct generalized advantage estimation from pairwise differences to obtain an unbiased policy-gradient estimator (R-GAE). Beyond theoretical results, we integrate RV with PPO and achieve competitive performance on the Atari benchmark (49 ALE games) compared to standard PPO, indicating that relative value estimation is an effective alternative to absolute critics.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 23 references that are in the map. First posted on arXiv 23 Jul 2026.

Builds on

15 of 23 references.

References · 15

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  9. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  10. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  11. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  12. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
3 more
  1. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  2. 2016Sample Efficient Actor-Critic with Experience Replaycited by 188
  3. 2021Direct Advantage Estimationcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Pretrain Value, Not Reward: Decoupled Value Policy Optimization
  2. 2023Conservative State Value Estimation for Offline Reinforcement Learning
  3. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning
  4. 2021Q-Value Weighted Regression: Reinforcement Learning with Limited Data
  5. 2026Optimal Value Inference for Reinforcement Learning
  6. 2024Statistical Inference for Policy Evaluation with Temporal Difference Learning
  7. 2026ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards
  8. 2026Inverting the Bellman Equation: From Q-Values to World Models