mlsanityData to 14 Oct 2026

Breaking Habits: On the Role of the Advantage Function in Learning Causal State Representations

arXiv 2506.11912 ↗First posted 13 Jun 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Recent work has shown that reinforcement learning agents can develop policies that exploit spurious correlations between rewards and observations. This phenomenon, known as policy confounding, arises because the agent's policy influences both past and future observation variables, creating a feedback loop that can hinder the agent's ability to generalize beyond its usual trajectories. In this paper, we show that the advantage function, commonly used in policy gradient methods, not only reduces the variance of gradient estimates but also mitigates the effects of policy confounding. By adjusting action values relative to the state representation, the advantage function downweights state-action pairs that are more likely under the current policy, breaking spurious correlations and encouraging the agent to focus on causal factors. We provide both analytical and empirical evidence demonstrating that training with the advantage function leads to improved out-of-trajectory performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 33 references that are in the map. First posted on arXiv 13 Jun 2025.

Builds on

19 of 33 references.

References · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2019Causal Confusion in Imitation Learningcited by 288
  3. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  4. 2018An Introduction to Deep Reinforcement Learningcited by 206
  5. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  6. 2017DARLA: Improving Zero-Shot Transfer in Reinforcement Learningcited by 149
  7. 2020Invariant Causal Prediction for Block MDPscited by 105
  8. 2019Observational Overfitting in Reinforcement Learningcited by 91
  9. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 82
  10. 2022Causal Imitation Learning with Unobserved Confounderscited by 41
  11. 2020Beyond variance reduction: Understanding the true impact of baselines on policy optimizationcited by 32
  12. 2023The Role of Baselines in Policy Gradient Optimizationcited by 25
7 more
  1. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
  2. 2021On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learningcited by 14
  3. 2021Direct Advantage Estimationcited by 11
  4. 2023Bad Habits: Policy Confounding and Out-of-Trajectory Generalization in RLcited by 10
  5. 2023Can Agents Run Relay Race with Strangers? Generalization of RL to Out-of-Distribution Trajectoriescited by 9
  6. 2024Skill or Luck? Return Decomposition via Advantage Functionscited by 8
  7. 2023What model does MuZero learn?cited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026An Informational Curse of Horizon in Goal-Conditioned Policy Learning
  2. 2026Reinforcement Learning via Advantage Guidance
  3. 2025Taming OOD Actions for Offline Reinforcement Learning: An Advantage-Based Approach
  4. 2025Model-Based Reinforcement Learning Under Confounding
  5. 2024Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions
  6. 2026On the "Causality" Step in Policy Gradient Derivations: A Pedagogical Reconciliation of Full Return and Reward-to-Go
  7. 2026Reinforcement Learning with Decomposed Subtasks
  8. 2026OPD+: Rethinking the Advantage Design for On-Policy Distillation