mlsanityData to 7 Oct 2026

Augmenting Self-attention with Persistent Memory

arXiv 1907.01470 ↗First posted 2 Jul 2019cs.LG cs.CL stat.MLcited by 92

Not on the map. Why?

Abstract (arXiv)

Transformer networks have lead to important progress in language modeling and machine translation. These models include two consecutive modules, a feed-forward layer and a self-attention layer. The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of Transformers. Building upon this intuition, we propose a new model that solely consists of attention layers. More precisely, we augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer. Thanks to these vectors, we can remove the feed-forward layer without degrading the performance of a transformer. Our evaluation shows the benefits brought by our model on standard character and word level language modeling benchmarks.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 16; 2023: 19; 2024: 20; 2025: 11; 2026: 13.

92 papers.

Sort:

By area:

  1. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  2. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  3. 2021A Survey of Transformerscited by 399
  4. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  5. 2024Titans: Learning to Memorize at Test Timecited by 303
  6. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  7. 2022Memorizing Transformerscited by 219
  8. 2022Analyzing Transformers in Embedding Spacecited by 131
  9. 2025Lumina-Image 2.0: A Unified and Efficient Image Generative Frameworkcited by 93
  10. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  11. 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
  12. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
38 more
  1. 2021Memory-efficient Transformers via Top-k Attentioncited by 59
  2. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  3. 2023A Study on ReLU and Softmax in Transformercited by 53
  4. 2022Universal Hopfield Networks: A General Framework for Single-Shot Associative Memory Modelscited by 52
  5. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  6. 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 45
  7. 2024WilKE: Wise-Layer Knowledge Editor for Lifelong Knowledge Editingcited by 40
  8. 2024Memory3: Language Modeling with Explicit Memorycited by 39
  9. 2022Retrieval-Augmented Transformer for Image Captioningcited by 38
  10. 2023Simplifying Transformer Blockscited by 37
  11. 2021When Attention Meets Fast Recurrence: Training Language Models with Reduced Computecited by 34
  12. 2021Not All Memories are Created Equal: Learning to Forget by Expiringcited by 32
  13. 2023Scaling Laws for Associative Memoriescited by 30
  14. 2021Attention Approximates Sparse Distributed Memorycited by 29
  15. 2021Mask Attention Networks: Rethinking and Strengthen Transformercited by 28
  16. 2021Improving Transformers with Probabilistic Attention Keyscited by 28
  17. 2023Transformer-VQ: Linear-Time Transformers via Vector Quantizationcited by 26
  18. 2023Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Modelscited by 23
  19. 2022Plug-and-Play Adaptation for Continuously-updated QAcited by 21
  20. 2022Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioningcited by 18
  21. 2023With a Little Help from your own Past: Prototypical Memory Networks for Image Captioningcited by 18
  22. 2024Memory Mosaicscited by 16
  23. 2022A survey on attention mechanisms for medical applications: are we moving towards better algorithms?cited by 15
  24. 2021Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modelingcited by 14
  25. 2022CaMEL: Mean Teacher Learning for Image Captioningcited by 14
  26. 2024Memorization in deep learning: A surveycited by 14
  27. 2024Beyond Scaling Laws: Understanding Transformer Performance with Associative Memorycited by 12
  28. 2024Distributional Associations vs In-Context Reasoning: A Study of Feed-forward and Attention Layerscited by 12
  29. 2024Human-inspired Perspectives: A Survey on AI Long-term Memorycited by 12
  30. 2025R3Mem: Bridging Memory Retention and Retrieval via Reversible Compressioncited by 12
  31. 2023Sparse Distributed Memory is a Continual Learnercited by 11
  32. 2024Position: An Inner Interpretability Framework for AI Inspired by Lessons from Cognitive Neurosciencecited by 11
  33. 2026Adaptive Loops and Memory in Transformers: Think Harder or Know More?cited by 11
  34. 2021RelTransformer: A Transformer-Based Long-Tail Visual Relationship Recognitioncited by 9
  35. 2023Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Modelcited by 9
  36. 2024Semformer: Transformer Language Models with Semantic Planningcited by 9
  37. 2024TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameterscited by 9
  38. 2025Learning to Insert [PAUSE] Tokens for Better Reasoningcited by 8

50 most cited of 92 All 92 (JSON)