mlsanityData to 7 Oct 2026

Memorizing Transformers

arXiv 2203.08913 ↗First posted 16 Mar 2022cs.LG cs.AI cs.CLcited by 219

Not on the map. Why?

Abstract (arXiv)

Language models typically need to be trained or finetuned in order to acquire new knowledge, which involves updating their weights. We instead envision language models that can simply read and memorize new data at inference time, thus acquiring new knowledge immediately. In this work, we extend language models with the ability to memorize the internal representations of past inputs. We demonstrate that an approximate kNN lookup into a non-differentiable memory of recent (key, value) pairs improves language modeling across various benchmarks and tasks, including generic webtext (C4), math papers (arXiv), books (PG-19), code (Github), as well as formal theorems (Isabelle). We show that the performance steadily improves when we increase the size of memory up to 262K tokens. On benchmarks including code and mathematics, we find that the model is capable of making use of newly defined functions and theorems during test time.

Key facts. Cited by 219 papers in the map (data through 7 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 16 Mar 2022.

Builds on

41 of 55 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2017Billion-scale similarity search with GPUscited by 2,477
  11. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
29 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2020Rethinking Attention with Performerscited by 1,584
  5. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  8. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  9. 2020Efficient Transformers: A Surveycited by 784
  10. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  11. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  12. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  13. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
  14. 2020Query-Key Normalization for Transformerscited by 266
  15. 2019Cross-Batch Memory for Embedding Learningcited by 130
  16. 2019Large Memory Layers with Product Keyscited by 120
  17. 2019Augmenting Self-attention with Persistent Memorycited by 92
  18. 2020Pre-training via Paraphrasingcited by 92
  19. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  20. 2021Adaptive Semiparametric Language Modelscited by 83
  21. 2016Improving Neural Language Models with a Continuous Cachecited by 81
  22. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64
  23. 2021Memory-efficient Transformers via Top-k Attentioncited by 59
  24. 2020Augmenting Transformers with KNN-Based Composite Memory for Dialoguecited by 53
  25. 2020Teaching Temporal Logics to Neural Networkscited by 51
  26. 2021H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequencescited by 47
  27. 2019Exploration of Neural Machine Translation in Autoformalization of Mathematics in Mizarcited by 40
  28. 2021Not All Memories are Created Equal: Learning to Forget by Expiringcited by 32
  29. 2021ReadTwice: Reading Very Large Documents with Memoriescited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 31; 2023: 61; 2024: 49; 2025: 31; 2026: 47.

219 papers.

Sort:

By area:

  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  4. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  5. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  6. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  7. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  8. 2022Confident Adaptive Language Modelingcited by 297
  9. 2022Long Range Language Modeling via Gated State Spacescited by 265
  10. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  11. 2022Autoformalization with Large Language Modelscited by 252
  12. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
38 more
  1. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  2. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  3. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  4. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  5. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  6. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  7. 2023Efficient Large Language Models: A Surveycited by 183
  8. 2023ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematicscited by 180
  9. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  10. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  11. 2022DocPrompting: Generating Code by Retrieving the Docscited by 151
  12. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 140
  13. 2023ChatDB: Augmenting LLMs with Databases as Their Symbolic Memorycited by 136
  14. 2022KNN-Diffusion: Image Generation via Large-Scale Retrievalcited by 135
  15. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  16. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 129
  17. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  18. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  19. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  20. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  21. 2022Training Language Models with Memory Augmentationcited by 112
  22. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  23. 2022Repository-Level Prompt Generation for Large Language Models of Codecited by 106
  24. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97
  25. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
  26. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  27. 2024Streaming Dense Video Captioningcited by 82
  28. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  29. 2022Efficient Transformers with Dynamic Token Poolingcited by 75
  30. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  31. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  32. 2022NaturalProver: Grounded Mathematical Proof Generation with Language Modelscited by 72
  33. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  34. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  35. 2023RET-LLM: Towards a General Read-Write Memory for Large Language Modelscited by 64
  36. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  37. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  38. 2022OpenCQA: Open-ended Question Answering with Chartscited by 60

50 most cited of 219 All 219 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024On Memorization of Large Language Models in Logical Reasoning
  2. 2025MLP Memory: A Retriever-Pretrained Memory for Large Language Models
  3. 2025Provable Benefits of In-Tool Learning for Large Language Models
  4. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
  5. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
  6. 2026Unlocking the Working Memory of Large Language Models for Latent Reasoning
  7. 2024Disentangling Memory and Reasoning Ability in Large Language Models
  8. 2026TransMem: Transforming Hidden States into Memory for Large Language Models