mlsanityData to 14 Oct 2026

Augmenting Language Models with Long-Term Memory

arXiv 2306.07174 ↗First posted 12 Jun 2023cs.CLcited by 189

Not on the map. Why?

Cited by 48 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing large language models (LLMs) can only afford fix-sized inputs due to the input length limit, preventing them from utilizing rich long-context information from past inputs. To address this, we propose a framework, Language Models Augmented with Long-Term Memory (LongMem), which enables LLMs to memorize long history. We design a novel decoupled network architecture with the original backbone LLM frozen as a memory encoder and an adaptive residual side-network as a memory retriever and reader. Such a decoupled memory design can easily cache and update long-term past contexts for memory retrieval without suffering from memory staleness. Enhanced with memory-augmented adaptation training, LongMem can thus memorize long past context and use long-term memory for language modeling. The proposed memory retrieval module can handle unlimited-length context in its memory bank to benefit various downstream tasks. Typically, LongMem can enlarge the long-form memory to 65k tokens and thus cache many-shot extra demonstration examples as long-form memory for in-context learning. Experiments show that our method outperforms strong long-context models on ChapterBreak, a challenging long-context modeling benchmark, and achieves remarkable improvements on memory-augmented in-context learning over LLMs. The results demonstrate that the proposed method is effective in helping language models to memorize and utilize long-form contents. Our code is open-sourced at https://aka.ms/LongMem.

Key facts. Cited by 189 papers in the map (data through 14 Oct 2026). Builds on 31 of its 40 references that are in the map. First posted on arXiv 12 Jun 2023.

Builds on

31 of 40 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
19 more
  1. 2020Longformer: The Long-Document Transformercited by 3,413
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  3. 2022Emergent Abilities of Large Language Modelscited by 2,512
  4. 2017Billion-scale similarity search with GPUscited by 2,497
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  12. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  13. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  14. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 230
  15. 2022Training Language Models with Memory Augmentationcited by 113
  16. 2022ChapterBreak: A Challenge Dataset for Long-Range Language Modelscited by 18
  17. 2022Visually-Augmented Language Modelingcited by 18
  18. 2021Task-Oriented Dialogue System as Natural Language Generationcited by 14
  19. 2023N-Gram Nearest Neighbor Machine Translationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 17; 2024: 52; 2025: 43; 2026: 77.

189 papers.

Sort:

By area:

  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  2. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  3. 2023A Comprehensive Overview of Large Language Modelscited by 648
  4. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  5. 2025Defeating Prompt Injections by Designcited by 297
  6. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  7. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  8. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  9. 2023Efficient Large Language Models: A Surveycited by 186
  10. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  11. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  12. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123
38 more
  1. 2024LLM Multi-Agent Systems: Challenges and Open Problemscited by 120
  2. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 119
  3. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  4. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  5. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 104
  6. 2024MEMORYLLM: Towards Self-Updatable Large Language Modelscited by 93
  7. 2023Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communicationcited by 82
  8. 2024EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Recordscited by 78
  9. 2024User-LLM: Efficient LLM Contextualization with User Embeddingscited by 76
  10. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  11. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  12. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 66
  13. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontierscited by 65
  14. 2023Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memorycited by 64
  15. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  16. 2025M+: Extending MemoryLLM with Scalable Long-Term Memorycited by 54
  17. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 48
  18. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  19. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approachescited by 43
  20. 2024Memory3: Language Modeling with Explicit Memorycited by 40
  21. 2023LLMR: Real-time Prompting of Interactive Worlds using Large Language Modelscited by 39
  22. 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agentscited by 37
  23. 2024Online Adaptation of Language Models with a Memory of Amortized Contextscited by 35
  24. 2024MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memorycited by 35
  25. 2024LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillationcited by 32
  26. 2024Investigate-Consolidate-Exploit: A General Strategy for Inter-Task Agent Self-Evolutioncited by 28
  27. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  28. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 28
  29. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  30. 2023Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruningcited by 26
  31. 2024Emotional RAG: Enhancing Role-Playing Agents through Emotional Retrievalcited by 26
  32. 2023Balancing Autonomy and Alignment: A Multi-Dimensional Taxonomy for Autonomous LLM-powered Multi-Agent Architecturescited by 25
  33. 2026M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactionscited by 25
  34. 2024CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memorycited by 24
  35. 2025CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactionscited by 22
  36. 2025Vision-centric Token Compression in Large Language Modelcited by 21
  37. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 20
  38. 2024Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Modelscited by 20

50 most cited of 189 All 189 (JSON)