mlsanityData to 7 Oct 2026

An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning

arXiv 2103.06224 ↗First posted 10 Mar 2021cs.LG cs.IT math.ITcited by 20

Not on the map. Why?

Abstract (arXiv)

How do we formalize the challenge of credit assignment in reinforcement learning? Common intuition would draw attention to reward sparsity as a key contributor to difficult credit assignment and traditional heuristics would look to temporal recency for the solution, calling upon the classic eligibility trace. We posit that it is not the sparsity of the reward itself that causes difficulty in credit assignment, but rather the information sparsity. We propose to use information theory to define this notion, which we then use to characterize when credit assignment is an obstacle to efficient learning. With this perspective, we outline several information-theoretic mechanisms for measuring credit under a fixed behavior policy, highlighting the potential of information theory as a key tool towards provably-efficient credit assignment.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 27 of its 75 references that are in the map. First posted on arXiv 10 Mar 2021.

Builds on

27 of 75 references.

References · 27

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  3. 2017Hindsight Experience Replaycited by 1,096
  4. 2018Exploration by Random Network Distillationcited by 966
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  8. 2016Deep Exploration via Bootstrapped DQNcited by 518
  9. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  10. 2016The Option-Critic Architecturecited by 433
  11. 2014An Information-Theoretic Analysis of Thompson Samplingcited by 178
  12. 2014Learning to Optimize via Information-Directed Samplingcited by 168
15 more
  1. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  2. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 162
  3. 2017Deep Exploration via Randomized Value Functionscited by 155
  4. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  5. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 76
  6. 2019Hindsight Credit Assignmentcited by 74
  7. 2018Optimizing Agent Behavior over Long Time Scales by Transporting Valuecited by 57
  8. 2019Information asymmetry in KL-regularized RLcited by 53
  9. 2018Sparse Attentive Backtracking: Temporal CreditAssignment Through Remindingcited by 33
  10. 2020Expected Eligibility Tracescited by 27
  11. 2019Separating value functions across time-scalescited by 15
  12. 2019Exploiting Hierarchy for Learning and Transfer in KL-regularized RLcited by 15
  13. 2017A Unified Bellman Equation for Causal Information and Value in Markov Decision Processescited by 11
  14. 2020Capacity of Continuous Channels with Memory via Directed Information Neural Estimatorcited by 5
  15. 2019Self-Attentional Credit Assignment for Transfer in Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 4; 2023: 5; 2024: 2; 2025: 2; 2026: 4.

20 papers.

Sort:

By area:

  1. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 56
  2. 2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
  3. 2025CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignmentcited by 25
  4. 2023Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysiscited by 10
  5. 2022Hierarchically Structured Task-Agnostic Continual Learningcited by 9
  6. 2026Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVRcited by 8
  7. 2021Mixture-of-Variational-Experts for Continual Learningcited by 7
  8. 2022The Importance of Credo in Multiagent Learningcited by 7
  9. 2022On Rate-Distortion Theory in Capacity-Limited Cognition & Reinforcement Learningcited by 6
  10. 2026Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Modelscited by 5
  11. 2022Planning to the Information Horizon of BAMDPs via Epistemic State Abstractioncited by 4
  12. 2023Hindsight-DICE: Stable Credit Assignment for Deep Reinforcement Learningcited by 4
8 more
  1. 2024Satisficing Exploration for Deep Reinforcement Learningcited by 4
  2. 2021Bad-Policy Density: A Measure of Reinforcement Learning Hardnesscited by 2
  3. 2023Towards a Better Understanding of Learning with Multiagent Teamscited by 2
  4. 2026Reflect-R1: Evidence-Driven Reflection for Self-Correction in Long Video Understandingcited by 2
  5. 2024Sequence Compression Speeds Up Credit Assignment in Reinforcement Learningcited by 1
  6. 2023Learning to Stabilize Online Reinforcement Learning in Unbounded State Spacescited by 0
  7. 2025Horizon Reduction as Information Loss in Offline Reinforcement Learningcited by 0
  8. 2026Counterfactual Rollout Replay: Forkable Environments as Free Process Rewards for Software Engineering Agentscited by 0