mlsanityData to 13 Oct 2026

Counterfactual Credit Assignment in Model-Free Reinforcement Learning

arXiv 2011.09464 ↗First posted 18 Nov 2020cs.LGcited by 61

Not on the map. Why?

Abstract (arXiv)

Credit assignment in reinforcement learning is the problem of measuring an action's influence on future rewards. In particular, this requires separating skill from luck, i.e. disentangling the effect of an action on rewards from that of external factors and subsequent actions. To achieve this, we adapt the notion of counterfactuals from causality theory to a model-free RL setup. The key idea is to condition value functions on future events, by learning to extract relevant information from a trajectory. We formulate a family of policy gradient algorithms that use these future-conditional value functions as baselines or critics, and show that they are provably low variance. To avoid the potential bias from conditioning on future information, we constrain the hindsight information to not contain information about the agent's actions. We demonstrate the efficacy and validity of our algorithm on a number of illustrative and challenging problems.

Key facts. Cited by 61 papers in the map (data through 13 Oct 2026). First posted on arXiv 18 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 9; 2022: 8; 2023: 14; 2024: 5; 2025: 9; 2026: 16.

61 papers.

Sort:

By area:

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,531
  2. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 118
  3. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  4. 2022MoCoDA: Model-based Counterfactual Data Augmentationcited by 44
  5. 2023Counterfactual Identifiability of Bijective Causal Modelscited by 42
  6. 2021Synthetic Returns for Long-Term Credit Assignmentcited by 27
  7. 2022Hindsight Learning for MDPs with Exogenous Inputscited by 25
  8. 2021Adaptive Rational Activations to Boost Deep Reinforcement Learningcited by 23
  9. 2024Interpretable Concept Bottlenecks to Align Reinforcement Learning Agentscited by 23
  10. 2023Causal Reinforcement Learning: A Surveycited by 22
  11. 2026Causal Agent Replay: Counterfactual Attribution for LLM-Agent Failurescited by 20
  12. 2023Large Sequence Models for Sequential Decision-Making: A Surveycited by 19
38 more
  1. 2023Counterfactual (Non-)identifiability of Learned Structural Causal Modelscited by 17
  2. 2023Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approachcited by 17
  3. 2022On the link between conscious function and general intelligence in humans and machinescited by 15
  4. 2023Partial Counterfactual Identification of Continuous Outcomes with a Curvature Sensitivity Modelcited by 14
  5. 2021Direct Advantage Estimationcited by 11
  6. 2023Transformers in Reinforcement Learning: A Surveycited by 11
  7. 2024Fine-Grained Causal Dynamics Learning with Quantization for Improving Robustness in Reinforcement Learningcited by 11
  8. 2023Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysiscited by 10
  9. 2022Actual Causality and Responsibility Attribution in Decentralized Partially Observable Markov Decision Processescited by 8
  10. 2023Semantically Aligned Task Decomposition in Multi-Agent Reinforcement Learningcited by 8
  11. 2024Skill or Luck? Return Decomposition via Advantage Functionscited by 8
  12. 2022Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When to Actcited by 7
  13. 2023Offline Imitation Learning with Variational Counterfactual Reasoningcited by 7
  14. 2024Assessing the Zero-Shot Capabilities of LLMs for Action Evaluation in RLcited by 7
  15. 2026Beyond Uniform Credit: Causal Credit Assignment for Policy Optimizationcited by 7
  16. 2021Policy Gradients Incorporating the Futurecited by 6
  17. 2021Towards Practical Credit Assignment for Deep Reinforcement Learningcited by 5
  18. 2024Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluationcited by 5
  19. 2026TeleBoost: A Systematic Alignment Framework for High-Fidelity, Controllable, and Robust Video Generationcited by 5
  20. 2022Selective Credit Assignmentcited by 4
  21. 2022Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environmentscited by 4
  22. 2023Hindsight-DICE: Stable Credit Assignment for Deep Reinforcement Learningcited by 4
  23. 2025Counterfactual reasoning: an analysis of in-context emergencecited by 4
  24. 2023PushWorld: A benchmark for manipulation planning with tools and movable obstaclescited by 3
  25. 2021Adaptive Pairwise Weights for Temporal Credit Assignmentcited by 2
  26. 2025Humans Coexist, So Must Embodied Artificial Agentscited by 2
  27. 2025Expediting Reinforcement Learning by Incorporating Knowledge About Temporal Causality in the Environmentcited by 2
  28. 2025Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformercited by 1
  29. 2026Tree-based Credit Assignment for Multi-Agent Memory Systemcited by 1
  30. 2026Counterfactual Quotient Models: Learning What Actions Change, Not What the World Doescited by 1
  31. 2026IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agentscited by 1
  32. 2021Free Will Belief as a consequence of Model-based Reinforcement Learningcited by 0
  33. 2022Shaping Advice in Deep Reinforcement Learningcited by 0
  34. 2023Policy Optimization with Smooth Guidance Learned from State-Only Demonstrationscited by 0
  35. 2025Learning Counterfactual Outcomes Under Rank Preservationcited by 0
  36. 2025What is a good matching of probability measures? A counterfactual lens on transport mapscited by 0
  37. 2025Counterfactual Credit Guided Bayesian Optimizationcited by 0
  38. 2025CFLight: Enhancing Safety with Traffic Signal Control through Counterfactual Learningcited by 0

50 most cited of 61 All 61 (JSON)