mlsanityData to 14 Oct 2026

Recurrent Memory Transformer

arXiv 2207.06881 ↗First posted 14 Jul 2022cs.CL cs.LGcited by 203

Not on the map. Why?

Abstract (arXiv)

Transformer-based models show their effectiveness across multiple domains and tasks. The self-attention allows to combine information from all sequence elements into context-aware representations. However, global and local information has to be stored mostly in the same element-wise representations. Moreover, the length of an input sequence is limited by quadratic computational complexity of self-attention. In this work, we propose and study a memory-augmented segment-level recurrent Transformer (RMT). Memory allows to store and process local and global information as well as to pass information between segments of the long sequence with the help of recurrence. We implement a memory mechanism with no changes to Transformer model by adding special memory tokens to the input or output sequence. Then the model is trained to control both memory operations and sequence representations processing. Results of experiments show that RMT performs on par with the Transformer-XL on language modeling for smaller memory sizes and outperforms it for tasks that require longer sequence processing. We show that adding memory tokens to Tr-XL is able to improve its performance. This makes Recurrent Memory Transformer a promising architecture for applications that require learning of long-term dependencies and general purpose in memory processing, such as algorithmic tasks and reasoning.

Key facts. Cited by 203 papers in the map (data through 14 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 14 Jul 2022.

Builds on

33 of 56 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  12. 2020Rethinking Attention with Performerscited by 1,601
21 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  3. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  4. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  5. 2014Neural Turing Machinescited by 644
  6. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  7. 2016Using Fast Weights to Attend to the Recent Pastcited by 170
  8. 2019Large Memory Layers with Product Keyscited by 124
  9. 2020MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioningcited by 79
  10. 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
  11. 2020Memory Transformercited by 76
  12. 2015Learning to Transduce with Unbounded Memorycited by 54
  13. 2020GMAT: Global Memory Augmentation for Transformerscited by 36
  14. 2020ETC: Encoding Long and Structured Inputs in Transformerscited by 34
  15. 2020ERNIE-Doc: A Retrospective Long-Document Modeling Transformercited by 31
  16. 2016Scaling Memory-Augmented Neural Networks with Sparse Reads and Writescited by 28
  17. 2017Memory Augmented Neural Networks with Wormhole Connectionscited by 15
  18. 2021Staircase Attention for Recurrent Processing of Sequencescited by 14
  19. 2016Dynamic Neural Turing Machine with Soft and Hard Addressing Schemescited by 8
  20. 2021Contrastive Document Representation Learning with Graph Attention Networkscited by 6
  21. 2022ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attentioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 39; 2024: 41; 2025: 38; 2026: 84.

203 papers.

Sort:

By area:

  1. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  2. 2023Vision Transformers Need Registerscited by 871
  3. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  4. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  5. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
  6. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  7. 2023Adapting Language Models to Compress Contextscited by 356
  8. 2024Titans: Learning to Memorize at Test Timecited by 328
  9. 2023Think before you speak: Training Language Models With Pause Tokenscited by 264
  10. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  11. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  12. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
38 more
  1. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  2. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  3. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  4. 2023Efficient Large Language Models: A Surveycited by 186
  5. 2025MIRIX: Multi-Agent Memory System for LLM-Based Agentscited by 173
  6. 2025Dynamic Cheatsheet: Test-Time Learning with Adaptive Memorycited by 163
  7. 2023ChatDB: Augmenting LLMs with Databases as Their Symbolic Memorycited by 140
  8. 2023Retrieval meets Long Context Large Language Modelscited by 115
  9. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  10. 2023LEGO-Prover: Neural Theorem Proving with Growing Librariescited by 109
  11. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 98
  12. 2024MEMORYLLM: Towards Self-Updatable Large Language Modelscited by 93
  13. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  14. 2023RecurrentGPT: Interactive Generation of (Arbitrarily) Long Textcited by 84
  15. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  16. 2024TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergycited by 71
  17. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontierscited by 65
  18. 2023Guiding Language Model Reasoning with Planning Tokenscited by 58
  19. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  20. 2025Mem-α: Learning Memory Construction via Reinforcement Learningcited by 55
  21. 2025M+: Extending MemoryLLM with Scalable Long-Term Memorycited by 54
  22. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 48
  23. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 47
  24. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 44
  25. 2023Compressed Context Memory For Online Language Model Interactioncited by 43
  26. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargementcited by 42
  27. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  28. 2024In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Misscited by 36
  29. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 36
  30. 2024MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memorycited by 35
  31. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  32. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  33. 2025ArcMemo: Abstract Reasoning Composition with Lifelong LLM Memorycited by 24
  34. 2024InfiniMotion: Mamba Boosts Memory in Transformer for Arbitrary Long Motion Generationcited by 23
  35. 2024TransformerFAM: Feedback attention is working memorycited by 22
  36. 2026EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policiescited by 22
  37. 2024Self-Updatable Large Language Models by Integrating Context into Model Parameterscited by 21
  38. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 21

50 most cited of 203 All 203 (JSON)