mlsanityData to 14 Oct 2026

Counterfactual Shapley Credit Assignment

arXiv 2607.16999 ↗First posted 18 Jul 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Interpretable and logic-guided RL

Abstract (arXiv)

The Credit Assignment Problem (CAP) is fundamental to developing efficient and explainable Reinforcement Learning (RL) agents. Existing frameworks, whether relying on temporal contiguity or hindsight-conditioned reward reweighting, frequently fail to attribute properly between an agent's policy (skill) and environmental stochasticity (luck). A principled approach to CAP must isolate the true causal drivers of observed outcomes from spurious correlations and environmental randomness. We introduce Counterfactual Shapley Credit Assignment, a novel framework grounded in causal theory that attributes credit and blame via the Counterfactual Shapley Value (φ-value). By redistributing environmental rewards, φ-values enhance temporal credit assignment across three critical dimensions: sparse causality, high stochasticity, and delayed rewards, all while preserving the optimal policy. We derive a consistent estimator that computes φ-values efficiently, enabling a new class of policy gradient methods, φ-PPO, combined with Prioritized Trajectory Replay (PTR). Empirical results demonstrate that φ-values align precisely to the ground truth causes of task rewards with superior sample efficiency in challenging environments where prior state-of-the-art methods fail to converge.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods. Builds on 23 of its 46 references that are in the map. First posted on arXiv 18 Jul 2026.

Builds on

23 of 46 references.

References · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  6. 2015Prioritized Experience Replaycited by 1,099
  7. 2019Solving Rubik's Cube with a Robot Handcited by 734
  8. 2017A Tutorial on Thompson Samplingcited by 433
  9. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  10. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 173
  11. 2020Improving KernelSHAP: Practical Shapley Value Estimation via Linear Regressioncited by 154
  12. 2019Hindsight Credit Assignmentcited by 79
11 more
  1. 2019A Survey of Deep Reinforcement Learning in Video Gamescited by 73
  2. 2020Counterfactual Credit Assignment in Model-Free Reinforcement Learningcited by 64
  3. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 59
  4. 2021Synthetic Returns for Long-Term Credit Assignmentcited by 27
  5. 2026DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignmentcited by 21
  6. 2023Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysiscited by 13
  7. 2026Intrinsic Credit Assignment for Long Horizon Interactioncited by 9
  8. 2020The Counterfactual NESS Definition of Causationcited by 8
  9. 2024Skill or Luck? Return Decomposition via Advantage Functionscited by 8
  10. 2021Modularity in Reinforcement Learning via Algorithmic Independence in Credit Assignmentcited by 6
  11. 2023Hindsight-DICE: Stable Credit Assignment for Deep Reinforcement Learningcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Explaining Reinforcement Learning: A Counterfactual Shapley Values Approach
  2. 2024Shapley Value Based Multi-Agent Reinforcement Learning: Theory, Method and Its Application to Energy Network
  3. 2023MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment
  4. 2026Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent System
  5. 2025Redistributing Rewards Across Time and Agents for Multi-Agent Reinforcement Learning
  6. 2026CRAFT: Counterfactual Credit Assignment from Free Sibling Rollouts for Self-Distilled Agentic Reinforcement Learning
  7. 2024Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning
  8. 2026Counterfactual Credit Policy Optimization for Multi-Agent Collaboration