mlsanityData to 14 Oct 2026

Scaling Transformer to 1M tokens and beyond with RMT

arXiv 2304.11062 ↗First posted 19 Apr 2023cs.CL cs.AI cs.LGcited by 98

Not on the map. Why?

Abstract (arXiv)

A major limitation for the broader scope of problems solvable by transformers is the quadratic scaling of computational complexity with input size. In this study, we investigate the recurrent memory augmentation of pre-trained transformer models to extend input context length while linearly scaling compute. Our approach demonstrates the capability to store information in memory for sequences of up to an unprecedented two million tokens while maintaining high retrieval accuracy. Experiments with language modeling tasks show perplexity improvement as the number of processed input segments increases. These results underscore the effectiveness of our method, which has significant potential to enhance long-term dependency handling in natural language understanding and generation tasks, as well as enable large-scale context processing for memory-intensive applications.

Key facts. Cited by 98 papers in the map (data through 14 Oct 2026). Builds on 33 of its 53 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

33 of 53 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2020Longformer: The Long-Document Transformercited by 3,413
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  10. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  11. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  12. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
21 more
  1. 2014Neural Turing Machinescited by 644
  2. 2014Memory Networkscited by 363
  3. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
  4. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  5. 2022Memorizing Transformerscited by 226
  6. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  7. 2022Recurrent Memory Transformercited by 203
  8. 2019The Lean mathematical librarycited by 134
  9. 2019Star-Transformercited by 94
  10. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  11. 2020MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioningcited by 79
  12. 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
  13. 2020Memory Transformercited by 76
  14. 2020Memformer: A Memory-Augmented Transformer for Sequence Modelingcited by 63
  15. 2015Learning to Transduce with Unbounded Memorycited by 54
  16. 2020GMAT: Global Memory Augmentation for Transformerscited by 36
  17. 2020ETC: Encoding Long and Structured Inputs in Transformerscited by 34
  18. 2020ERNIE-Doc: A Retrospective Long-Document Modeling Transformercited by 31
  19. 2016Scaling Memory-Augmented Neural Networks with Sparse Reads and Writescited by 28
  20. 2017Memory Augmented Neural Networks with Wormhole Connectionscited by 15
  21. 2016Dynamic Neural Turing Machine with Soft and Hard Addressing Schemescited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 32; 2024: 42; 2025: 15; 2026: 9.

98 papers.

Sort:

By area:

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  4. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  5. 2024Titans: Learning to Memorize at Test Timecited by 328
  6. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 246
  7. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  8. 2023Language Modeling Is Compressioncited by 225
  9. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  10. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  11. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  12. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
38 more
  1. 2023ChatDB: Augmenting LLMs with Databases as Their Symbolic Memorycited by 140
  2. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 133
  3. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  4. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123
  5. 2023LEGO-Prover: Neural Theorem Proving with Growing Librariescited by 109
  6. 2024Long Context Compression with Activation Beaconcited by 104
  7. 2024AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agentscited by 102
  8. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  9. 2024From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMscited by 79
  10. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  11. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  12. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 69
  13. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 68
  14. 2025M+: Extending MemoryLLM with Scalable Long-Term Memorycited by 54
  15. 2026EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoningcited by 54
  16. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 50
  17. 2026RoboMME: Benchmarking and Understanding Memory for Robotic Generalist Policiescited by 49
  18. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 48
  19. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 44
  20. 2024Foundation Models for Music: A Surveycited by 37
  21. 2024In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Misscited by 36
  22. 2024Mixture of In-Context Prompters for Tabular PFNscited by 33
  23. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  24. 2023Interactive Natural Language Processingcited by 31
  25. 2024NetSafe: Exploring the Topological Safety of Multi-agent Networkscited by 29
  26. 2024LongGenBench: Long-context Generation Benchmarkcited by 28
  27. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  28. 2024GRAM: Global Reasoning for Multi-Page VQAcited by 23
  29. 2024InfiniMotion: Mamba Boosts Memory in Transformer for Arbitrary Long Motion Generationcited by 23
  30. 2024Extending LLMs' Context Window with 100 Samplescited by 21
  31. 2024Self-Updatable Large Language Models by Integrating Context into Model Parameterscited by 21
  32. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20
  33. 2025U-NIAH: Unified RAG and LLM Evaluation for Long Context Needle-In-A-Haystackcited by 20
  34. 2024COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Trainingcited by 18
  35. 2023TextGenSHAP: Scalable Post-hoc Explanations in Text Generation with Long Documentscited by 17
  36. 2024Towards LifeSpan Cognitive Systemscited by 13
  37. 2024Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-context Modelscited by 13
  38. 2023Focus Your Attention (with Adaptive IIR Filters)cited by 12

50 most cited of 98 All 98 (JSON)