mlsanityData to 14 Oct 2026

Direct Advantage Estimation

arXiv 2109.06093 ↗First posted 13 Sep 2021cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

The predominant approach in reinforcement learning is to assign credit to actions based on the expected return. However, we show that the return may depend on the policy in a way which could lead to excessive variance in value estimation and slow down learning. Instead, we show that the advantage function can be interpreted as causal effects and shares similar properties with causal representations. Based on this insight, we propose Direct Advantage Estimation (DAE), a novel method that can model the advantage function and estimate it directly from on-policy data while simultaneously minimizing the variance of the return without requiring the (action-)value function. We also relate our method to Temporal Difference methods by showing how value functions can be seamlessly integrated into DAE. The proposed method is easy to implement and can be readily adapted by modern actor-critic methods. We evaluate DAE empirically on three discrete control domains and show that it can outperform generalized advantage estimation (GAE), a strong baseline for advantage estimation, on a majority of the environments when applied to policy optimization.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 16 of its 42 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

16 of 42 references.

References · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  9. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  10. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  11. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  12. 2016Sample Efficient Actor-Critic with Experience Replaycited by 188
4 more
  1. 2019Off-Policy Evaluation in Partially Observable Environmentscited by 66
  2. 2020Counterfactual Credit Assignment in Model-Free Reinforcement Learningcited by 64
  3. 2013Graded Causation and Defaultscited by 13
  4. 2020Disentangling causal effects for hierarchical reinforcement learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 1; 2026: 4.

11 papers.

  1. 2023VA-learning as a more efficient alternative to Q-learningcited by 11
  2. 2024Skill or Luck? Return Decomposition via Advantage Functionscited by 8
  3. 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learningcited by 7
  4. 2023Decision-Aware Actor-Critic with Function Approximation and Theoretical Guaranteescited by 3
  5. 2023Can Active Sampling Reduce Causal Confusion in Offline Reinforcement Learning?cited by 2
  6. 2026On the Variance of Temporal Difference Learning and its Reduction Using Control Variatescited by 1
  7. 2026Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learningcited by 1
  8. 2024Disentangled Representations for Causal Cognitioncited by 0
  9. 2025Breaking Habits: On the Role of the Advantage Function in Learning Causal State Representationscited by 0
  10. 2026Relative Value Learningcited by 0
  11. 2026ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewardscited by 0