mlsanityData to 7 Oct 2026

When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment

arXiv 2307.03864 ↗First posted 7 Jul 2023cs.LGcited by 62

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) algorithms face two distinct challenges: learning effective representations of past and present observations, and determining how actions influence future returns. Both challenges involve modeling long-term dependencies. The Transformer architecture has been very successful to solve problems that involve long-term dependencies, including in the RL domain. However, the underlying reason for the strong performance of Transformer-based RL methods remains unclear: is it because they learn effective memory, or because they perform effective credit assignment? After introducing formal definitions of memory length and credit assignment length, we design simple configurable tasks to measure these distinct quantities. Our empirical results reveal that Transformers can enhance the memory capability of RL algorithms, scaling up to tasks that require memorizing observations 1500 steps ago. However, Transformers do not improve long-term credit assignment. In summary, our results provide an explanation for the success of Transformers in RL, while also highlighting an important area for future research and benchmark design. Our code is open-sourced at https://github.com/twni2016/Memory-RL

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 43 of its 71 references that are in the map. First posted on arXiv 7 Jul 2023.

Builds on

43 of 71 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  11. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  12. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
31 more
  1. 2022Transformers are Sample-Efficient World Modelscited by 317
  2. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  3. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  4. 2022Multi-Game Decision Transformerscited by 212
  5. 2022Online Decision Transformercited by 194
  6. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 162
  7. 2019Soft Actor-Critic for Discrete Action Settingscited by 161
  8. 2014Difference Target Propagationcited by 144
  9. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  10. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 137
  11. 2019Behaviour Suite for Reinforcement Learningcited by 129
  12. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  13. 2019Meta reinforcement learning as task inferencecited by 75
  14. 2013How to Construct Deep Recurrent Neural Networkscited by 64
  15. 2020Counterfactual Credit Assignment in Model-Free Reinforcement Learningcited by 60
  16. 2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 59
  17. 2018Optimizing Agent Behavior over Long Time Scales by Transporting Valuecited by 57
  18. 2021Memory-based Deep Reinforcement Learning for POMDPscited by 53
  19. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  20. 2019Variational Recurrent Models for Solving Partially Observable Control Taskscited by 46
  21. 2022Evaluating Long-Term Memory in 3D Mazescited by 41
  22. 2022Transformers are Meta-Reinforcement Learnerscited by 39
  23. 2021Towards mental time travel: a hierarchical memory for reinforcement learning agentscited by 38
  24. 2019Generalization of Reinforcement Learners with Working and Episodic Memorycited by 35
  25. 2022Deep Transformer Q-Networks for Partially Observable Reinforcement Learningcited by 35
  26. 2018Sparse Attentive Backtracking: Temporal CreditAssignment Through Remindingcited by 33
  27. 2021Learning Long-Term Reward Redistribution via Randomized Return Decompositioncited by 33
  28. 2021Acting in Delayed Environments with Non-Stationary Markov Policiescited by 28
  29. 2019Sequence Modeling of Temporal Credit Assignment for Episodic Reinforcement Learningcited by 27
  30. 2021Synthetic Returns for Long-Term Credit Assignmentcited by 27
  31. 2021Recurrent Off-policy Baselines for Memory-based Continuous Controlcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 26; 2025: 15; 2026: 18.

62 papers.

Sort:

By area:

  1. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  2. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  3. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learningcited by 46
  4. 2024Mastering Memory Tasks with World Modelscited by 38
  5. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  6. 2025Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learningcited by 34
  7. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 20
  8. 2024Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modelingcited by 18
  9. 2024TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learningcited by 18
  10. 2023Memory-Consistent Neural Networks for Imitation Learningcited by 14
  11. 2024A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Taskscited by 14
  12. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
38 more
  1. 2024UniZero: Generalized and Efficient Planning with Scalable Latent World Modelscited by 12
  2. 2025Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformerscited by 12
  3. 2025FindingDory: A Benchmark to Evaluate Memory in Embodied Agentscited by 11
  4. 2024Do Transformer World Models Give Better Policy Gradients?cited by 10
  5. 2024Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Ratecited by 10
  6. 2024Real-Time Recurrent Learning using Trace Units in Reinforcement Learningcited by 10
  7. 2024Rethinking Transformers in Solving POMDPscited by 9
  8. 2024Agent-state based policies in POMDPs: Beyond belief-state MDPscited by 9
  9. 2025Memo: Training Memory-Efficient Embodied Agents with Reinforcement Learningcited by 9
  10. 2024Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learningcited by 8
  11. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  12. 2024Learning to Play Atari in a World of Tokenscited by 6
  13. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformercited by 6
  14. 2025Benchmarking Partial Observability in Reinforcement Learning with a Suite of Memory-Improvable Domainscited by 6
  15. 2025ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problemscited by 6
  16. 2024Recurrent Reinforcement Learning with Memoroidscited by 5
  17. 2024Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observabilitycited by 5
  18. 2024Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluationcited by 5
  19. 2026Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Actioncited by 4
  20. 2024An Empirical Study on the Power of Future Prediction in Partially Observable Environmentscited by 3
  21. 2024ReLIC: A Recipe for 64k Steps of In-Context Reinforcement Learning for Embodied AIcited by 3
  22. 2025Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environmentscited by 3
  23. 2024Reinforcement Learning via Auxiliary Task Distillationcited by 2
  24. 2025RLBenchNet: The Right Network for the Right Reinforcement Learning Taskcited by 2
  25. 2024Sequence Compression Speeds Up Credit Assignment in Reinforcement Learningcited by 1
  26. 2024AdaCred: Adaptive Causal Decision Transformers with Feature Creditingcited by 1
  27. 2025Investigating Memory in Model-Free RL with POPGym Arcadecited by 1
  28. 2025The challenge of hidden gifts in multi-agent reinforcement learningcited by 1
  29. 2025Recurrent State Encoders for Efficient Neural Combinatorial Optimizationcited by 1
  30. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  31. 2026Memory Retention Is Not Enough to Master Memory Tasks in Reinforcement Learningcited by 1
  32. 2026Perception-Based Beliefs for POMDPs with Visual Observationscited by 1
  33. 2026Hierarchical Reinforcement Learning with Augmented Step-Level Transitions for LLM Agentscited by 1
  34. 2026HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learningcited by 1
  35. 2025What Do You Need for Compositional Generalization in Diffusion Planning?cited by 0
  36. 2025A New Perspective on Transformers in Online Reinforcement Learning for Continuous Controlcited by 0
  37. 2025Learning to Focus: Prioritizing Informative Histories with Structured Attention Mechanisms in Partially Observable Reinforcement Learningcited by 0
  38. 2026Meta-Learning and Meta-Reinforcement Learning -- Tracing the Path towards DeepMind's Adaptive Agentcited by 0

50 most cited of 62 All 62 (JSON)