mlsanityData to 14 Oct 2026

Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learning

arXiv 2606.20411 ↗First posted 18 Jun 2026cs.LGcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Direct Advantage Estimation (DAE) has been shown to improve the sample efficiency of deep reinforcement learning algorithms. However, its reliance on full environment observability limits its applicability in realistic settings, and its requirement to model transition probabilities incurs substantial computational overhead for high-dimensional observations. In the present work, we address both limitations. First, we extend the theoretical framework of DAE to partially observable domains with minimal modifications. Second, we reduce its computational complexity by introducing discrete latent dynamics models that efficiently approximate transition probabilities. We evaluate our approach on the Arcade Learning Environment and find that DAE scales effectively with function approximator capacity while retaining high sample efficiency.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 49 of its 77 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

49 of 77 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Skill or Luck? Return Decomposition via Advantage Functions20 Feb

Other references · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2016Layer Normalizationcited by 5,669
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2017Neural Discrete Representation Learningcited by 4,712
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  8. 2018World Modelscited by 1,513
  9. 2019Root Mean Square Layer Normalizationcited by 1,480
  10. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  11. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  12. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
36 more
  1. 2021Scaling Vision Transformerscited by 999
  2. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  4. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  5. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  6. 2017Deep Reinforcement Learning that Matterscited by 754
  7. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  8. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  9. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  10. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  11. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  12. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  13. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  14. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  15. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  16. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  17. 2015Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networkscited by 139
  18. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  19. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
  20. 2016Learning in an Uncertain World: Representing Ambiguity Through Multiple Hypothesescited by 97
  21. 2019Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Predictioncited by 96
  22. 2019Deep Generalized Method of Moments for Instrumental Variable Analysiscited by 88
  23. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 78
  24. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representationscited by 74
  25. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 62
  26. 2019Variational Recurrent Models for Solving Partially Observable Control Taskscited by 47
  27. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  28. 2019Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the DQN Targetcited by 31
  29. 2021Procedural Generalization by Planning with Self-Supervised World Modelscited by 31
  30. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
  31. 2017Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognitioncited by 19
  32. 2021Direct Advantage Estimationcited by 11
  33. 2023VA-learning as a more efficient alternative to Q-learningcited by 11
  34. 2024FlashRNN: I/O-Aware Optimization of Traditional RNNs on modern hardwarecited by 7
  35. 2019Rethinking Full Connectivity in Recurrent Neural Networkscited by 6
  36. 2024Geometry-Aware Instrumental Variable Regressioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Search-Based Adversarial Estimates for Improving Sample Efficiency in Off-Policy Reinforcement Learning
  2. 2025Generalized Advantage Estimation for Distributional Policy Gradients
  3. 2026On Advantage Estimates for Max@K Policy Gradients
  4. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
  5. 2025Benchmarking Partial Observability in Reinforcement Learning with a Suite of Memory-Improvable Domains
  6. 2025Taming OOD Actions for Offline Reinforcement Learning: An Advantage-Based Approach
  7. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  8. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective