mlsanityData to 7 Oct 2026

Recurrent Action Transformer with Memory

arXiv 2306.09459 ↗First posted 15 Jun 2023cs.LG cs.AIcited by 17

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Transformers have become increasingly popular in offline reinforcement learning (RL) due to their ability to treat agent trajectories as sequences, reframing policy learning as a sequence modeling task. However, in partially observable environments (POMDPs), effective decision-making depends on retaining information about past events -- something that standard transformers struggle with due to the quadratic complexity of self-attention, which limits their context length. One solution to this problem is to extend transformers with memory mechanisms. We propose the Recurrent Action Transformer with Memory (RATE), a novel transformer-based architecture for offline RL that incorporates a recurrent memory mechanism designed to regulate information retention. We evaluate RATE across a diverse set of environments: memory-intensive tasks (ViZDoom-Two-Colors, T-Maze, Memory Maze, Minigrid-Memory, and POPGym), as well as standard Atari and MuJoCo benchmarks. Our comprehensive experiments demonstrate that RATE significantly improves performance in memory-dependent settings while remaining competitive on standard tasks across a broad range of baselines. These findings underscore the pivotal role of integrated memory mechanisms in offline RL and establish RATE as a unified, high-capacity architecture for effective decision-making over extended horizons. Code: https://sites.google.com/view/rate-model/.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 63 of its 92 references that are in the map. First posted on arXiv 15 Jun 2023.

Builds on

63 of 92 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Decision S4: Efficient Sequence-Based RL via State Spaces Layers8 Jun
  2. 2022How Crucial is Transformer in Decision Transformer?26 Nov
  3. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL8 Sep
  4. 2022Multi-Game Decision Transformers30 May
  5. 2022Online Decision Transformer11 Feb
  6. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
38 more
  1. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  2. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2020GLU Variants Improve Transformercited by 1,724
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  10. 2020Rethinking Attention with Performerscited by 1,584
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  13. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  14. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  15. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  16. 2022A Generalist Agentcited by 802
  17. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  18. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  19. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  20. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  21. 2016Learning to reinforcement learncited by 369
  22. 2022Transformers are Sample-Efficient World Modelscited by 317
  23. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  24. 2022Memorizing Transformerscited by 219
  25. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  26. 2022Recurrent Memory Transformercited by 197
  27. 2020SceneFormer: Indoor Scene Generation with Transformerscited by 154
  28. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  29. 2022On The Computational Complexity of Self-Attentioncited by 137
  30. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 137
  31. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  32. 2020MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioningcited by 79
  33. 2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 59
  34. 2018Learning Longer-term Dependencies in RNNs with Auxiliary Lossescited by 58
  35. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  36. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  37. 2022Evaluating Long-Term Memory in 3D Mazescited by 41
  38. 2022Transformers are Meta-Reinforcement Learnerscited by 39

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 6.

17 papers.

Sort:

By area:

  1. 2025Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learningcited by 34
  2. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  3. 2025SRMT: Shared Memory for Multi-agent Lifelong Pathfindingcited by 8
  4. 2026μVLA: On Recurrent Memory for Partially Observable Manipulation in VLA Modelscited by 7
  5. 2024Learning Memory Mechanisms for Decision Making through Demonstrationscited by 6
  6. 2025ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problemscited by 6
  7. 2024Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluationcited by 5
  8. 2025DAM-Seg: Anatomically accurate cardiac segmentation using Dense Associative Networkscited by 3
  9. 2025Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environmentscited by 3
  10. 2025Re:Frame -- Retrieving Experience From Associative Memorycited by 1
  11. 2025Do We Need Transformers to Play FPS Video Games?cited by 0
  12. 2025Accelerating Transformers in Online RLcited by 0
5 more
  1. 2026MARL-GPT: Foundation Model for Multi-Agent Reinforcement Learningcited by 0
  2. 2026Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillationcited by 0
  3. 2026ALER: Adaptive Learnable Experience Rewriting for Reinforcement Learningcited by 0
  4. 2026Decision Titan: Test-Time Training for Long-Term Memory in Offline Reinforcement Learningcited by 0
  5. 2026Self-Supervised Keyframe Discovery for Horizon-Invariant Behavior Cloningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Multi-Objective Decision Transformers for Offline Reinforcement Learning
  2. 2025Decision SpikeFormer: Spike-Driven Transformer for Decision Making
  3. 2022Contextual Transformer for Offline Meta Reinforcement Learning
  4. 2022Bootstrapped Transformer for Offline Reinforcement Learning
  5. 2024Solving Continual Offline Reinforcement Learning with Decision Transformer
  6. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learning
  7. 2025Mental Accounts for Actions: EWA-Inspired Attention in Decision Transformers
  8. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RL