mlsanityData to 14 Oct 2026

Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures

arXiv 2508.10824 ↗First posted 14 Aug 2025cs.LG cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Memory is fundamental to intelligence, enabling learning, reasoning, and adaptability across biological and artificial systems. While Transformer architectures excel at sequence modeling, they face critical limitations in long-range context retention, continual learning, and knowledge integration. This review presents a unified framework bridging neuroscience principles, including dynamic multi-timescale memory, selective attention, and consolidation, with engineering advances in Memory-Augmented Transformers. We organize recent progress through three taxonomic dimensions: functional objectives (context extension, reasoning, knowledge integration, adaptation), memory representations (parameter-encoded, state-based, explicit, hybrid), and integration mechanisms (attention fusion, gated control, associative retrieval). Our analysis of core memory operations (reading, writing, forgetting, and capacity management) reveals a shift from static caches toward adaptive, test-time learning systems. We identify persistent challenges in scalability and interference, alongside emerging solutions including hierarchical buffering and surprise-gated updates. This synthesis provides a roadmap toward cognitively-inspired, lifelong-learning Transformer architectures.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 71 of its 123 references that are in the map. First posted on arXiv 14 Aug 2025.

Builds on

71 of 123 references.

References · 71

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  5. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  7. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  8. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  9. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
  10. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  11. 2014Neural Turing Machinescited by 644
  12. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
38 more
  1. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  2. 2024Titans: Learning to Memorize at Test Timecited by 328
  3. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 281
  4. 2022Transformer Memory as a Differentiable Search Indexcited by 247
  5. 2022Memorizing Transformerscited by 226
  6. 2023Augmenting Language Models with Long-Term Memorycited by 189
  7. 2025Hierarchical Reasoning Modelcited by 177
  8. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 171
  9. 2025Memory OS of AI Agentcited by 150
  10. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 133
  11. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  12. 2025From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMscited by 124
  13. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  14. 2024MEMORYLLM: Towards Self-Updatable Large Language Modelscited by 93
  15. 2023MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Trackingcited by 88
  16. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
  17. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 66
  18. 2023Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memorycited by 64
  19. 2020Memformer: A Memory-Augmented Transformer for Sequence Modelingcited by 63
  20. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  21. 2025M+: Extending MemoryLLM with Scalable Long-Term Memorycited by 54
  22. 2024Self-evolving Agents with reflective and memory-augmented abilitiescited by 52
  23. 2024Memory Layers at Scalecited by 48
  24. 2025SCORE: Story Coherence and Retrieval Enhancement for AI Narrativescited by 42
  25. 2024Memory3: Language Modeling with Explicit Memorycited by 40
  26. 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Modelscited by 37
  27. 2021ABC: Attention with Bounded-memory Controlcited by 36
  28. 2022An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Taskscited by 36
  29. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 36
  30. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  31. 2024MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memorycited by 35
  32. 2020Self-Attentive Associative Memorycited by 33
  33. 2025Cognitive Memory in Large Language Modelscited by 29
  34. 2023Adaptive Computation with Elastic Input Sequencecited by 22
  35. 2024TransformerFAM: Feedback attention is working memorycited by 22
  36. 2024HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processingcited by 21
  37. 2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decodingcited by 21
  38. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 44
  2. 2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
  3. 2025Context-Aware Multi-Turn Visual-Textual Reasoning in LVLMs via Dynamic Memory and Adaptive Visual Guidancecited by 1
  4. 2025Memory as Resonance: A Biomimetic Architecture for Infinite Context Memory on Ergodic Phonetic Manifoldscited by 1
  5. 2026Hierarchical Chain-of-Thought: Enhancing LLM Reasoning Performance and Efficiencycited by 1
  6. 2026Where Should Experience Live? Hierarchical Hebbian Memory for Continual Vision Transformerscited by 1
  7. 2026TRAM: Enhancing Multimodal Reasoning with Trajectory-Derived Auxiliary Memorycited by 0
  8. 2026From Observation to Intervention: Memory in Brains and Large Language Modelscited by 0
  9. 2026Memory as a cache: Exact context reuse and deletion by constructioncited by 0