mlsanityData to 14 Oct 2026

KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows

arXiv 2507.07400 ↗First posted 10 Jul 2025cs.DC cs.MAcited by 56

On the map: LLM agents and tool use → Software engineering coding agents

also in 1 more
Abstract (arXiv)

Large language model (LLM) based agentic workflows have become a popular paradigm for coordinating multiple specialized agents to solve complex tasks. To improve serving efficiency, existing LLM systems employ prefix caching to reuse key-value (KV) tensors corresponding to agents' fixed prompts, thereby avoiding redundant computation across repeated invocations. However, current systems typically evict KV caches using a Least Recently Used (LRU) policy, which fails to anticipate future agent usage and often discards KV caches shortly before their reuse. This leads to frequent cache misses and substantial recomputation or swapping overhead. We present KVFlow, a workflow-aware KV cache management framework tailored for agentic workloads. KVFlow abstracts the agent execution schedule as an Agent Step Graph and assigns each agent a steps-to-execution value that estimates its temporal proximity to future activation. These values guide a fine-grained eviction policy at the KV node level, allowing KVFlow to preserve entries likely to be reused and efficiently manage shared prefixes in tree-structured caches. Moreover, KVFlow introduces a fully overlapped KV prefetching mechanism, which proactively loads required tensors from CPU to GPU in background threads for agents scheduled in the next step, thereby avoiding cache miss stalls during generation. Compared to SGLang with hierarchical radix cache, KVFlow achieves up to 1.83× speedup for single workflows with large prompts, and up to 2.19× speedup for scenarios with many concurrent workflows.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 33 of its 43 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

33 of 43 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programs19 Feb
  2. 2025Adaptive Self-improvement LLM Agentic System for ML Library Development4 Feb
  3. 2025Argos: Agentic Time-Series Anomaly Detection with Autonomous Rule Generation via Large Language Models24 Jan
  4. 2024AFlow: Automating Agentic Workflow Generation14 Oct
  5. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation16 Aug
  6. 2023ChatDev: Communicative Agents for Software Development16 Jul

Other references · 27

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  8. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  10. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  11. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
15 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  2. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 251
  3. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  4. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  5. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 141
  6. 2024AgentScope: A Flexible yet Robust Multi-Agent Platformcited by 121
  7. 2024Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineeringcited by 103
  8. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  9. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 98
  10. 2023Stateful Large Language Model Serving with Pensievecited by 61
  11. 2024Andes: Defining and Enhancing Quality-of-Experience in LLM-Based Text Streaming Servicescited by 45
  12. 2024InferCept: Efficient Intercept Support for Augmented Large Language Model Inferencecited by 33
  13. 2024Very Large-Scale Multi-Agent Simulation in AgentScopecited by 18
  14. 2025Cognify: Supercharging Gen-AI Workflows With Hierarchical Autotuningcited by 7
  15. 2024PEER: Expertizing Domain-Specific Tasks with a Multi-Agent Framework and Tuning Methodscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 52.

56 papers.

Sort:

By area:

  1. 2026Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Servingcited by 20
  2. 2026Beyond Max Tokens: Stealthy Resource Amplification via Tool Calling Chains in LLM Agentscited by 14
  3. 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspectivecited by 11
  4. 2026LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agentscited by 10
  5. 2026Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devicescited by 8
  6. 2026ICaRus: Identical Cache Reuse for Efficient Multi Model Inferencecited by 8
  7. 2025Batch Query Processing and Optimization for Agentic Workflowscited by 6
  8. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Servingcited by 6
  9. 2026SAGA: Workflow-Atomic Scheduling for AI Agent Inference on GPU Clusterscited by 6
  10. 2026LCGuard: Latent Communication Guard for Safe KV Sharing in Multi-Agent Systemscited by 6
  11. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  12. 2026Sutradhara: An Intelligent Orchestrator-Engine Co-design for Tool-based Agentic Inferencecited by 5
38 more
  1. 2026KEEP: A KV-Cache-Centric Memory Management System for Efficient Embodied Planningcited by 5
  2. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 4
  3. 2026PrefillShare: A Shared Prefill Module for KV Reuse in Multi-LLM Disaggregated Servingcited by 4
  4. 2026Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Managementcited by 4
  5. 2026Learning What Not to Forget: Long-Horizon Agent Memory from a Few Kilobytes of Learningcited by 4
  6. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  7. 2026KV Cache Translation across Heterogeneous Large Language Modelscited by 4
  8. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  9. 2026Beyond Accuracy: Unveiling Inefficiency Patterns in Tool-Integrated Reasoningcited by 3
  10. 2026PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inferencecited by 3
  11. 2026Not All Tokens Are Worth Caching: Learning Semantic-Aware Eviction for LLM Prefix Cachescited by 3
  12. 2026Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulationcited by 3
  13. 2026MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inferencecited by 3
  14. 2026When KV Cache Reuse Fails in Multi-Agent Systems: Cross-Candidate Interaction is Crucial for LLM Judgescited by 2
  15. 2026ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Managementcited by 2
  16. 2026ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cachecited by 2
  17. 2026A Policy-Driven Runtime Layer for Agentic LLM Servingcited by 2
  18. 2026VikingMem: A Memory Base Management System for Stateful LLM-based Applicationscited by 2
  19. 2026Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Servingcited by 2
  20. 2026Speculative Macro Commit for Faster Tool-Using Agentscited by 2
  21. 2026GORGO: Online Tuning for Cross-Region Network-Aware LLM Servingcited by 1
  22. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inferencecited by 1
  23. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restorationcited by 1
  24. 2026PRISM: Fast Online LLM Serving via Scheduling-Memory Co-designcited by 1
  25. 2026Benchmarking LLM Serving Systems for Agentic AI Workloads with XPerfcited by 1
  26. 2026TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Servingcited by 1
  27. 2026Taking Tokenization off the Host for Agentic LLM Servingcited by 1
  28. 2026AMMA: A Multi-Chiplet Memory-Centric Architecture for Low-Latency 1M Context Attention Servingcited by 0
  29. 2026Streaming Communication in Multi-Agent Reasoningcited by 0
  30. 2026SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineeringcited by 0
  31. 2026PEEK: Predictive Queue-Informed KV Cache Management for LLM Servingcited by 0
  32. 2026Workload-Aware Caching for Multi-Agent Systemscited by 0
  33. 2026StepReflect: Structured UI Transition Reflection for Mobile GUI Agentscited by 0
  34. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0
  35. 2026BIO-MEMART: Biometric-Aware KV Cache Memory for Multi-User LLM Agentscited by 0
  36. 2026Dynamic HBM Repartitioning for Multi-Turn MoE Servingcited by 0
  37. 2026PipeSwift: Revisiting Pipeline Parallelism for Large-Scale Completion-Oriented Agentic LLM Servingcited by 0
  38. 2026The KV Cache Working Set: Online Capacity Planning for LLM Inference Systemscited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning Agent Execution for KV-Cache Management in Agentic Serving
  2. 2026[AAFLOW+] Stateful Operator Abstraction with Zero-Copy Distributed KV Cache Orchestration for Multi-Agent Workflows
  3. 2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharing
  4. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
  5. 2025KVCOMM: Online Cross-context KV-cache Communication for Efficient LLM-based Multi-agent Systems
  6. 2026ReCache: Efficient KV Cache Reuse and Compression for Tool-Augmented LLM Agents
  7. 2025Accelerating Language Model Workflows with Prompt Choreography
  8. 2026CacheWise: Understanding Workloads and Optimizing KVCache Management for Efficiently Serving LLM Coding Agents