mlsanityData to 7 Oct 2026

Cognitive Memory in Large Language Models

arXiv 2504.02441 ↗First posted 3 Apr 2025cs.CL cs.AIcited by 28

Not on the map. Why?

Abstract (arXiv)

This paper examines memory mechanisms in Large Language Models (LLMs), emphasizing their importance for context-rich responses, reduced hallucinations, and improved efficiency. It categorizes memory into sensory, short-term, and long-term, with sensory memory corresponding to input prompts, short-term memory processing immediate context, and long-term memory implemented via external databases or structures. The text-based memory section covers acquisition (selection and summarization), management (updating, accessing, storing, and resolving conflicts), and utilization (full-text search, SQL queries, semantic search). The KV cache-based memory section discusses selection methods (regularity-based summarization, score-based approaches, special token embeddings) and compression techniques (low-rank compression, KV merging, multimodal compression), along with management strategies like offloading and shared attention mechanisms. Parameter-based memory methods (LoRA, TTT, MoE) transform memories into model parameters to enhance efficiency, while hidden-state-based memory approaches (chunk mechanisms, recurrent transformers, Mamba model) improve long-text processing by combining RNN hidden states with current methods. Overall, the paper offers a comprehensive analysis of LLM memory mechanisms, highlighting their significance and future research directions.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 71 of its 88 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

71 of 88 references.

References · 71

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2020Longformer: The Long-Document Transformercited by 3,362
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2020Reformer: The Efficient Transformercited by 1,658
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  9. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 539
  10. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  11. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  12. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
38 more
  1. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  2. 2023Adapting Language Models to Compress Contextscited by 350
  3. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  4. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  5. 2022Test-Time Training with Masked Autoencoderscited by 208
  6. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  7. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  8. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  9. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 140
  10. 2021Learned Token Pruning for Transformerscited by 138
  11. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  12. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  13. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  14. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
  15. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 99
  16. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 94
  17. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
  18. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 87
  19. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  20. 2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 82
  21. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 81
  22. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  23. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matterscited by 73
  24. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  25. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  26. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  27. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  28. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
  29. 2023Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memorycited by 64
  30. 2023Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Modelscited by 62
  31. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 61
  32. 2023Max-Margin Token Selection in Attention Mechanismcited by 57
  33. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  34. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  35. 2024Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengthscited by 44
  36. 2023Compressed Context Memory For Online Language Model Interactioncited by 42
  37. 2024Compress to Impress: Unleashing the Potential of Compressive Memory in Real-World Long-Term Conversationscited by 36
  38. 2024Effectively Compress KV Heads for LLMcited by 30

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 17; 2026: 11.

28 papers.

Sort:

By area:

  1. 2025MIRIX: Multi-Agent Memory System for LLM-Based Agentscited by 160
  2. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  3. 2025MemOS: A Memory OS for AI Systemcited by 129
  4. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  5. 2025MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Modelscited by 68
  6. 2025MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systemscited by 56
  7. 2025Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Managementcited by 51
  8. 2025Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflectscited by 40
  9. 2026HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understandingcited by 36
  10. 2025StoryBench: A Dynamic Benchmark for Evaluating Long-Term Memory with Multi Turnscited by 20
  11. 2026HiMem: Hierarchical Long-Term Memory for LLM Long-Horizon Agentscited by 16
  12. 2025MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environmentscited by 13
16 more
  1. 2025Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architecturescited by 9
  2. 2025Understanding the Thinking Process of Reasoning Models: A Perspective from Schoenfeld's Episode Theorycited by 9
  3. 2025Smarter Together: Creating Agentic Communities of Practice through Shared Experiential Learningcited by 5
  4. 2026MSSR: Memory-Aware Adaptive Replay for Continual LLM Fine-Tuningcited by 4
  5. 2026Your Code Agent Can Grow Alongside You with Structured Memorycited by 4
  6. 2026ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RLcited by 3
  7. 2025The Paradox of Poetic Intent in Back-Translation: Evaluating the Quality of Large Language Models in Chinese Translationcited by 2
  8. 2025GDGS: 3D Gaussian Splatting Via Geometry-Guided Initialization And Dynamic Density Controlcited by 2
  9. 2025Beyond Training: Enabling Self-Evolution of Agents with MOBIMEMcited by 2
  10. 2025A Global-Local Cross-Attention Network for Ultra-high Resolution Remote Sensing Image Semantic Segmentationcited by 1
  11. 2026CraniMem: Cranial Inspired Gated and Bounded Memory for Agentic Systemscited by 1
  12. 2026SuperLocalMemory V3.3: The Living Brain -- Biologically-Inspired Forgetting, Cognitive Quantization, and Multi-Channel Retrieval for Zero-LLM Agent Memory Systemscited by 1
  13. 2026Who Does This Name Remind You of ? Nationality Prediction via Large Language Model Associative Memorycited by 0
  14. 2026Context Recycling for Long-Horizon LLM Inferencecited by 0
  15. 2026MAViE: A Multi-scale Adaptive Vision Encoder for Fine-grained Visual Perception and Efficient Multimodal Reasoningcited by 0
  16. 2026StreamFlow: Dynamic Memory Flows for Streaming Video Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolution
  2. 2024On the Structural Memory of LLM Agents
  3. 2023SCM: Enhancing Large Language Model with Self-Controlled Memory Framework
  4. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  5. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
  6. 2023SoK: Memorization in General-Purpose Large Language Models
  7. 2024A Survey on Large Language Model Acceleration based on KV Cache Management
  8. 2024A critical review of methods and challenges in large language models