mlsanityData to 7 Oct 2026

Pancake: Hierarchical Memory System for Multi-Agent LLM Serving

arXiv 2602.21477 ↗First posted 25 Feb 2026cs.MAcited by 5

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

In this work, we identify and address the core challenges of agentic memory management in LLM serving, where large-scale storage, frequent updates, and multiple coexisting agents jointly introduce complex and high-cost approximate nearest neighbor (ANN) searching problems. We present Pancake, a multi-tier agentic memory system that unifies three key techniques: (i) multi-level index caching for single agents, (ii) coordinated index management across multiple agents, and (iii) collaborative GPU-CPU acceleration. Pancake exposes easy-to-use interface that can be integrated into memory-based agents like Mem-GPT, and is compatible with agentic frameworks such as LangChain and LlamaIndex. Experiments on realistic agent workloads show that Pancake substantially outperforms existing frameworks, achieving more than 4.29x end-to-end throughput improvement.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 54 of its 84 references that are in the map. First posted on arXiv 25 Feb 2026.

Builds on

54 of 84 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory29 Sep
  2. 2025G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems9 Jun
  3. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory28 Apr
  4. 2025A-MEM: Agentic Memory for LLM Agents17 Feb
  5. 2024A Survey on the Memory Mechanism of Large Language Model based Agents21 Apr

Other references · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  12. 2017Billion-scale similarity search with GPUscited by 2,477
37 more
  1. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  3. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  4. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  5. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  6. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  7. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  8. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  9. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,002
  13. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  14. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  15. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 539
  16. 2023Active Retrieval Augmented Generationcited by 506
  17. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  18. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  19. 2024The Faiss librarycited by 370
  20. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  21. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  22. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  23. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  24. 2024HoneyComb: A Flexible LLM-Based Agent System for Materials Sciencecited by 56
  25. 2023A Comprehensive Survey on Vector Database: Storage and Retrieval Technique, Challengecited by 55
  26. 2021FreshDiskANN: A Fast and Accurate Graph-Based ANN Index for Streaming Similarity Searchcited by 36
  27. 2024PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-designcited by 28
  28. 2024SPFresh: Incremental In-Place Update for Billion-Scale Vector Searchcited by 28
  29. 2024Accelerating Retrieval-Augmented Language Model Serving with Speculationcited by 20
  30. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 16
  31. 2024BANG: Billion-Scale Approximate Nearest Neighbor Search using a Single GPUcited by 14
  32. 2025DeepPersona: A Generative Engine for Scaling Deep Synthetic Personascited by 14
  33. 2025TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrievalcited by 9
  34. 2025Quake: Adaptive Indexing for Vector Searchcited by 8
  35. 2024Enhancing HNSW Index for Real-Time Updates: Addressing Unreachable Points and Performance Degradationcited by 7
  36. 2024Incremental IVF Index Maintenance for Streaming Vector Searchcited by 7
  37. 2025LSM-VEC: A Large-Scale Disk-Based System for Dynamic Vector Searchcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026HARBOR: Automated Harness Optimizationcited by 9
  2. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Servingcited by 6
  3. 2026Eywa: Provenance-Grounded Long-Term Memory for AI Agentscited by 3
  4. 2026SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineeringcited by 0
  5. 2026PrimeScientist: Strategic Allocation of Research Effort in Autonomous Researchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
  2. 2026Multi-Agent Memory from a Computer Architecture Perspective: Visions and Challenges Ahead
  3. 2026MemForest: An Efficient Agent Memory System with Hierarchical Temporal Indexing
  4. 2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing
  5. 2026AMV-L: Lifecycle-Managed Agent Memory for Tail-Latency Control in Long-Running LLM Systems
  6. 2026Are We Ready For An Agent-Native Memory System?
  7. 2026Filesystem-Based Memory for LLM Agents: Organization, Evolution, and Sustainability
  8. 2026ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Management