mlsanityData to 7 Oct 2026

A Survey of Temporal Credit Assignment in Deep Reinforcement Learning

arXiv 2312.01072 ↗First posted 2 Dec 2023cs.LG cs.AIcited by 56

Not on the map. Why?

Abstract (arXiv)

The Credit Assignment Problem (CAP) refers to the longstanding challenge of Reinforcement Learning (RL) agents to associate actions with their long-term consequences. Solving the CAP is a crucial step towards the successful deployment of RL in the real world since most decision problems provide feedback that is noisy, delayed, and with little or no information about the causes. These conditions make it hard to distinguish serendipitous outcomes from those caused by informed decision-making. However, the mathematical nature of credit and the CAP remains poorly understood and defined. In this survey, we review the state of the art of Temporal Credit Assignment (CA) in deep RL. We propose a unifying formalism for credit that enables equitable comparisons of state-of-the-art algorithms and improves our understanding of the trade-offs between the various methods. We cast the CAP as the problem of learning the influence of an action over an outcome from a finite amount of experience. We discuss the challenges posed by delayed effects, transpositions, and a lack of action influence, and analyse how existing methods aim to address them. Finally, we survey the protocols to evaluate a credit assignment method and suggest ways to diagnose the sources of struggle for different methods. Overall, this survey provides an overview of the field for new-entry practitioners and researchers, it offers a coherent perspective for scholars looking to expedite the starting stages of a new study on the CAP, and it suggests potential directions for future research.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 35 of its 220 references that are in the map. First posted on arXiv 2 Dec 2023.

Builds on

35 of 220 references.

References · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2015Prioritized Experience Replaycited by 1,070
  4. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  5. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  6. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  7. 2022Multi-Game Decision Transformerscited by 212
  8. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  9. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 158
  10. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  11. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  12. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
23 more
  1. 2020Chip Placement with Deep Reinforcement Learningcited by 83
  2. 2016DeepMind Labcited by 74
  3. 2018Relational Deep Reinforcement Learningcited by 72
  4. 2021A Survey of Exploration Methods in Reinforcement Learningcited by 68
  5. 2020Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentationcited by 57
  6. 2021Bootstrapped Meta-Learningcited by 45
  7. 2019Reward Shaping via Meta-Learningcited by 30
  8. 2022Adversarial Policies Beat Superhuman Go AIscited by 29
  9. 2020Policy Evaluation Networkscited by 28
  10. 2022The Phenomenon of Policy Churncited by 28
  11. 2022Human-level Atari 200x fastercited by 28
  12. 2021An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learningcited by 20
  13. 2018Forward-Backward Reinforcement Learningcited by 17
  14. 2015Emphatic Temporal-Difference Learningcited by 16
  15. 2019Adversarial Exploitation of Policy Imitationcited by 13
  16. 2022Upside-Down Reinforcement Learning Can Diverge in Stochastic Environments With Episodic Resetscited by 10
  17. 2022All You Need Is Supervised Learning: From Imitation Learning to Meta-RL With Upside Down RLcited by 9
  18. 2020A Self-Tuning Actor-Critic Algorithmcited by 6
  19. 2015Learning to Predict Independent of Spancited by 5
  20. 2020Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewardscited by 4
  21. 2022Selective Credit Assignmentcited by 4
  22. 2022How Does Return Distribution in Distributional Reinforcement Learning Help Optimization?cited by 4
  23. 2022Learning Relative Return Policies With Upside-Down Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 23; 2026: 27.

56 papers.

Sort:

By area:

  1. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  2. 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?cited by 105
  3. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
  4. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 77
  5. 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoningcited by 49
  6. 2025First Return, Entropy-Eliciting Explorecited by 45
  7. 2026DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignmentcited by 20
  8. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Modelscited by 19
  9. 2026Intelligent AI Delegationcited by 16
  10. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actionscited by 15
  11. 2026Mem-T: Densifying Rewards for Long-Horizon Memory Agentscited by 14
  12. 2025Towards Improving Reward Design in RL: A Reward Alignment Metric for RL Practitionerscited by 10
38 more
  1. 2025Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editingcited by 9
  2. 2025Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimizationcited by 9
  3. 2024Assessing the Zero-Shot Capabilities of LLMs for Action Evaluation in RLcited by 7
  4. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  5. 2025DSADF: Thinking Fast and Slow for Decision Makingcited by 6
  6. 2025TextAtari: 100K Frames Game Playing with Language Agentscited by 6
  7. 2025Align to Structure: Aligning Large Language Models with Structural Informationcited by 5
  8. 2025AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learningcited by 5
  9. 2026Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energycited by 5
  10. 2026Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillationcited by 5
  11. 2025Beyond Outcome Reward: Decoupling Search and Answering Improves LLM Agentscited by 4
  12. 2026Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVRcited by 4
  13. 2024Action abstractions for amortized samplingcited by 3
  14. 2025Self-Correcting Code Generation Using Small Language Modelscited by 3
  15. 2026Where Did It Go Wrong? Process-Level Evaluation of Web Agents with Semantic State Trackingcited by 3
  16. 2025Humans Coexist, So Must Embodied Artificial Agentscited by 2
  17. 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learningcited by 2
  18. 2026Not all tokens are needed(NAT): token efficient reinforcement learningcited by 2
  19. 2026GRAIL: Gradient-Reweighted Advantages for Reinforcement Learning with Verifiable Rewardscited by 2
  20. 2026Reflect-R1: Evidence-Driven Reflection for Self-Correction in Long Video Understandingcited by 2
  21. 2026Don't Solve, Just Compare: Tiny Advisors for Runtime Intervention in LLM Agentscited by 2
  22. 2024Sequence Compression Speeds Up Credit Assignment in Reinforcement Learningcited by 1
  23. 2024Scaffolded Language Models with Language Supervision for Mixed-Autonomy: A Surveycited by 1
  24. 2025Deep Reinforcement Learning for Dynamic Origin-Destination Matrix Estimation in Microscopic Traffic Simulations Considering Credit Assignmentcited by 1
  25. 2026Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agentscited by 1
  26. 2026EDIT: Evidence-Diagnosed Intervention Training for Rule-Faithful LLM Gradingcited by 1
  27. 2026EXPO-SQL: Execution-based Clause-level Policy Optimization for Text-to-SQLcited by 1
  28. 2026Learning from Environmental Feedback: Credit Assignment across Multiple Timescales for Agentic Reinforcement Learningcited by 1
  29. 2025Reinforcement Learning with Graph Attention for Routing and Wavelength Assignment with Lightpath Reusecited by 0
  30. 2025Reasoning without Regretcited by 0
  31. 2025Credit Assignment and Efficient Exploration based on Influence Scope in Multi-agent Reinforcement Learningcited by 0
  32. 2025ToMPO: Training LLM Strategic Decision Making from a Multi-Agent Perspectivecited by 0
  33. 2025A Markov Decision Process for Variable Selection in Branch & Boundcited by 0
  34. 2026Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinitycited by 0
  35. 2026Biased Error Attribution in Multi-Agent Human-AI Systems Under Delayed Feedbackcited by 0
  36. 2026Timescale Separation Enables Deep Reinforcement Learning Control of Rotating Detonation Engine Mode Transitionscited by 0
  37. 2026LC-ERD: Mining Latent Logic for Self-Evolving Reasoning via Consistency-Regulated Reward Decompositioncited by 0
  38. 2026Horizon Adaptive Offline Policy Learning via Value Stitchingcited by 0

50 most cited of 56 All 56 (JSON)