mlsanityData to 14 Oct 2026

Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning

arXiv 2405.03878 ↗First posted 6 May 2024cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Temporal credit assignment in reinforcement learning is challenging due to delayed and stochastic outcomes. Monte Carlo targets can bridge long delays between action and consequence but lead to high-variance targets due to stochasticity. Temporal difference (TD) learning uses bootstrapping to overcome variance but introduces a bias that can only be corrected through many iterations. TD(λ) provides a mechanism to navigate this bias-variance tradeoff smoothly. Appropriately selecting λ can significantly improve performance. Here, we propose Chunked-TD, which uses predicted probabilities of transitions from a model for computing λ-return targets. Unlike other model-based solutions to credit assignment, Chunked-TD is less vulnerable to model inaccuracies. Our approach is motivated by the principle of history compression and 'chunks' trajectories for conventional TD learning. Chunking with learned world models compresses near-deterministic regions of the environment-policy interaction to speed up credit assignment while still bootstrapping when necessary. We propose algorithms that can be implemented online and show that they solve some problems much faster than conventional TD(λ).

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 24 of its 44 references that are in the map. First posted on arXiv 6 May 2024.

Builds on

24 of 44 references.

References · 24

  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2020Mastering Atari with Discrete World Modelscited by 1,007
  3. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  4. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  5. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 162
  6. 2018Meta-Gradient Reinforcement Learningcited by 159
  7. 2019Behaviour Suite for Reinforcement Learningcited by 132
  8. 2019When to use parametric models in reinforcement learning?cited by 118
  9. 2019Hindsight Credit Assignmentcited by 79
  10. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 63
  11. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 59
  12. 2017Hybrid Reward Architecture for Reinforcement Learningcited by 58
12 more
  1. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 47
  2. 2016Self-Correcting Models for Model-Based Reinforcement Learningcited by 43
  3. 2020Selective Dyna-style Planning Under Limited Model Capacitycited by 30
  4. 2020Expected Eligibility Tracescited by 27
  5. 2018Recall Traces: Backtracking Models for Efficient Reinforcement Learningcited by 23
  6. 2016A Greedy Approach to Adapting the Trace Parameter for Temporal Difference Learningcited by 20
  7. 2021An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learningcited by 20
  8. 2020Forethought and Hindsight in Credit Assignmentcited by 16
  9. 2023Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysiscited by 13
  10. 2019Adaptive Temporal-Difference Learning for Policy Evaluation with Per-State Uncertainty Estimatescited by 6
  11. 2021Towards Practical Credit Assignment for Deep Reinforcement Learningcited by 5
  12. 2019Source Traces for Temporal Difference Learningcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Action abstractions for amortized samplingcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2025Stabilizing Temporal Difference Learning via Implicit Stochastic Recursion
  2. 2025Reinforcement Learning with Action Chunking
  3. 2024Demystifying the Recency Heuristic in Temporal-Difference Learning
  4. 2023Temporal Difference Learning with Compressed Updates: Error-Feedback meets Reinforcement Learning
  5. 2025Is Temporal Difference Learning the Gold Standard for Stitching in RL?
  6. 2026Towards Parameter-Free Temporal Difference Learning
  7. 2026Counterfactual Shapley Credit Assignment
  8. 2026Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning