# Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures

arXiv: https://arxiv.org/abs/2508.10824 · HTML: https://mlsanity.com/paper/2508.10824/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 71 of its 123 references that are in the map.
- First posted on arXiv 14 Aug 2025.

## Builds on (most cited first, 30 of 71)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [Titans: Learning to Memorize at Test Time](https://mlsanity.com/paper/2501.00663/) (2024)
- [HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models](https://mlsanity.com/paper/2405.14831/) (2024)
- [Transformer Memory as a Differentiable Search Index](https://mlsanity.com/paper/2202.06991/) (2022)
- [Memorizing Transformers](https://mlsanity.com/paper/2203.08913/) (2022)
- [Augmenting Language Models with Long-Term Memory](https://mlsanity.com/paper/2306.07174/) (2023)
- [Hierarchical Reasoning Model](https://mlsanity.com/paper/2506.21734/) (2025)
- [JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models](https://mlsanity.com/paper/2311.05997/) (2023)
- [Memory OS of AI Agent](https://mlsanity.com/paper/2506.06326/) (2025)
- [Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading](https://mlsanity.com/paper/2310.05029/) (2023)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs](https://mlsanity.com/paper/2504.15965/) (2025)
- [Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference](https://mlsanity.com/paper/2403.09636/) (2024)
- [MEMORYLLM: Towards Self-Updatable Large Language Models](https://mlsanity.com/paper/2402.04624/) (2024)
- [MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Tracking](https://mlsanity.com/paper/2307.15700/) (2023)
- [Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models](https://mlsanity.com/paper/2402.02244/) (2024)
- [ATLAS: Learning to Optimally Memorize the Context at Test Time](https://mlsanity.com/paper/2505.23735/) (2025)
- [Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memory](https://mlsanity.com/paper/2311.08719/) (2023)

## Built on by (most cited first, 9 of 9)

- [A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents](https://mlsanity.com/paper/2602.06052/) (2026)
- [MeSH: Memory-as-State-Highways for Recursive Transformers](https://mlsanity.com/paper/2510.07739/) (2025)
- [Context-Aware Multi-Turn Visual-Textual Reasoning in LVLMs via Dynamic Memory and Adaptive Visual Guidance](https://mlsanity.com/paper/2509.05669/) (2025)
- [Memory as Resonance: A Biomimetic Architecture for Infinite Context Memory on Ergodic Phonetic Manifolds](https://mlsanity.com/paper/2512.20245/) (2025)
- [Hierarchical Chain-of-Thought: Enhancing LLM Reasoning Performance and Efficiency](https://mlsanity.com/paper/2604.00130/) (2026)
- [Where Should Experience Live? Hierarchical Hebbian Memory for Continual Vision Transformers](https://mlsanity.com/paper/2609.00358/) (2026)
- [TRAM: Enhancing Multimodal Reasoning with Trajectory-Derived Auxiliary Memory](https://mlsanity.com/paper/2608.01922/) (2026)
- [From Observation to Intervention: Memory in Brains and Large Language Models](https://mlsanity.com/paper/2608.12377/) (2026)
- [Memory as a cache: Exact context reuse and deletion by construction](https://mlsanity.com/paper/2609.32395/) (2026)

## Abstract (arXiv)

Memory is fundamental to intelligence, enabling learning, reasoning, and adaptability across biological and artificial systems. While Transformer architectures excel at sequence modeling, they face critical limitations in long-range context retention, continual learning, and knowledge integration. This review presents a unified framework bridging neuroscience principles, including dynamic multi-timescale memory, selective attention, and consolidation, with engineering advances in Memory-Augmented Transformers. We organize recent progress through three taxonomic dimensions: functional objectives (context extension, reasoning, knowledge integration, adaptation), memory representations (parameter-encoded, state-based, explicit, hybrid), and integration mechanisms (attention fusion, gated control, associative retrieval). Our analysis of core memory operations (reading, writing, forgetting, and capacity management) reveals a shift from static caches toward adaptive, test-time learning systems. We identify persistent challenges in scalability and interference, alongside emerging solutions including hierarchical buffering and surprise-gated updates. This synthesis provides a roadmap toward cognitively-inspired, lifelong-learning Transformer architectures.