mlsanityData to 14 Oct 2026

Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis

arXiv 2306.16803 ↗First posted 29 Jun 2023cs.LG stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

To make reinforcement learning more sample efficient, we need better credit assignment methods that measure an action's influence on future rewards. Building upon Hindsight Credit Assignment (HCA), we introduce Counterfactual Contribution Analysis (COCOA), a new family of model-based credit assignment algorithms. Our algorithms achieve precise credit assignment by measuring the contribution of actions upon obtaining subsequent rewards, by quantifying a counterfactual query: 'Would the agent still have reached this reward if it had taken another action?'. We show that measuring contributions w.r.t. rewarding states, as is done in HCA, results in spurious estimates of contributions, causing HCA to degrade towards the high-variance REINFORCE estimator in many relevant environments. Instead, we measure contributions w.r.t. rewards or learned representations of the rewarding objects, resulting in gradient estimates with lower variance. We run experiments on a suite of problems specifically designed to evaluate long-term credit assignment capabilities. By using dynamic programming, we measure ground-truth policy gradients and show that the improved performance of our new model-based credit assignment methods is due to lower bias and variance compared to HCA and common baselines. Our results demonstrate how modeling action contributions towards rewarding outcomes can be leveraged for credit assignment, opening a new path towards sample-efficient reinforcement learning.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 51 of its 94 references that are in the map. First posted on arXiv 29 Jun 2023.

Builds on

51 of 94 references.

References · 51

  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  6. 2023Mastering Diverse Domains through World Modelscited by 1,253
  7. 2017Hindsight Experience Replaycited by 1,136
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  9. 2020Mastering Atari with Discrete World Modelscited by 1,007
  10. 2018Exploration by Random Network Distillationcited by 993
  11. 2016Deep Variational Information Bottleneckcited by 930
  12. 2019Solving Rubik's Cube with a Robot Handcited by 734
38 more
  1. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  2. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  3. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  4. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 219
  5. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 173
  6. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
  7. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  8. 2020Estimating Counterfactual Treatment Outcomes over Time Through Adversarially Balanced Representationscited by 155
  9. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 125
  10. 2015Gradient Estimation Using Stochastic Computation Graphscited by 121
  11. 2019Training Agents using Upside-Down Reinforcement Learningcited by 107
  12. 2018Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Searchcited by 82
  13. 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 81
  14. 2019Hindsight Credit Assignmentcited by 79
  15. 2018Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselinescited by 68
  16. 2019Shaping Belief States with Generative Environment Models for RLcited by 66
  17. 2020Counterfactual Credit Assignment in Model-Free Reinforcement Learningcited by 64
  18. 2018Optimizing Agent Behavior over Long Time Scales by Transporting Valuecited by 58
  19. 2018Temporal Difference Variational Auto-Encodercited by 49
  20. 2018The Mirage of Action-Dependent Baselines in Reinforcement Learningcited by 48
  21. 2019Is the Policy Gradient a Gradient?cited by 40
  22. 2020Reinforcement Learning with Trajectory Feedbackcited by 40
  23. 2017Hindsight policy gradientscited by 37
  24. 2021Self-Supervised Online Reward Shaping in Sparse-Reward Environmentscited by 37
  25. 2020Align-RUDDER: Learning From Few Demonstrations by Reward Redistributioncited by 36
  26. 2018Sparse Attentive Backtracking: Temporal CreditAssignment Through Remindingcited by 34
  27. 2021Learning Long-Term Reward Redistribution via Randomized Return Decompositioncited by 33
  28. 2020Expected Eligibility Tracescited by 27
  29. 2021Synthetic Returns for Long-Term Credit Assignmentcited by 27
  30. 2018Recall Traces: Backtracking Models for Efficient Reinforcement Learningcited by 23
  31. 2017Policy Gradient Methods for Reinforcement Learning with Function Approximation and Action-Dependent Baselinescited by 20
  32. 2021An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learningcited by 20
  33. 2019Credit Assignment Techniques in Stochastic Computation Graphscited by 14
  34. 2020Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Modelscited by 14
  35. 2020Value-driven Hindsight Modellingcited by 6
  36. 2021Policy Gradients Incorporating the Futurecited by 6
  37. 2021Towards Practical Credit Assignment for Deep Reinforcement Learningcited by 5
  38. 2021InferNet for Delayed Reinforcement Tasks: Addressing the Temporal Credit Assignment Problemcited by 2

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 10.

13 papers.

Sort:

By area:

  1. 2026Self-Evolving Agents with Anytime-Valid Certificatescited by 6
  2. 2026Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillationcited by 4
  3. 2026Counterfactual Quotient Models: Learning What Actions Change, Not What the World Doescited by 2
  4. 2024Sequence Compression Speeds Up Credit Assignment in Reinforcement Learningcited by 1
  5. 2026Tree-based Credit Assignment for Multi-Agent Memory Systemcited by 1
  6. 2025Learning Counterfactual Outcomes Under Rank Preservationcited by 0
  7. 2025Counterfactual Credit Guided Bayesian Optimizationcited by 0
  8. 2026Counterfactual Shapley Credit Assignmentcited by 0
  9. 2026PACT: From Credit Assignment to Critic Alignmentcited by 0
  10. 2026Dependency-Aware Reward Shaping for Agentic Reinforcement Learningcited by 0
  11. 2026Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learningcited by 0
  12. 2026Of Errors and Echoes: Responsibility-Aware Credit Assignment for Tool-Use Agentscited by 0
1 more
  1. 2026CoCA: Criterion-Aware Credit Assignment for Rubric-Based Reinforcement Learningcited by 0