mlsanityData to 7 Oct 2026

DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference

arXiv 2602.21548 ↗First posted 25 Feb 2026cs.DCcited by 15

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The performance of multi-turn, agentic LLM inference is increasingly dominated by KV-Cache storage I/O rather than computation. In prevalent disaggregated architectures, loading the massive KV-Cache from external storage creates a fundamental imbalance: storage NICs on prefill engines become bandwidth-saturated, while those on decoding engines remain idle. This asymmetry severely constrains overall system throughput. We present DualPath, an inference system that breaks this bottleneck by introducing dual-path KV-Cache loading. Beyond the traditional storage-to-prefill path, DualPath enables a novel storage-to-decode path, in which the KV-Cache is loaded into decoding engines and then efficiently transferred to prefill engines via RDMA over the compute network. DualPath combines this optimized data path -- which inherently avoids network congestion and avoids interference with latency-critical model execution communications -- with a global scheduler that dynamically balances load across prefill and decode engines. Our evaluation on three models with production agentic workloads demonstrates that DualPath improves offline inference throughput by up to 1.87× on our in-house inference system. It can also improve online serving throughput by an average factor of 1.96× without violating SLO.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 50 references that are in the map. First posted on arXiv 25 Feb 2026.

Builds on

24 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  5. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  6. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  11. 2024A Survey on Large Language Models for Code Generationcited by 646
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
10 more
  1. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  2. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  3. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturescited by 38
  4. 2024Fast State Restoration in LLM Serving with HCachecited by 32
  5. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  6. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  7. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 13
  8. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputationcited by 12
  9. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimizationcited by 10
  10. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Servingcited by 14
  2. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Servingcited by 6
  3. 2026Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Managementcited by 4
  4. 2026Stage-Aware Communication Scheduling for Disaggregated LLM Servingcited by 1
  5. 2026TENT: A Declarative Slice Spraying Engine for Performant and Resilient Data Movement in Disaggregated LLM Servingcited by 1
  6. 2026A Token/KV-Cache Communication Media Selection and Resource Allocation Strategy for Multi-Agent Collaborationcited by 1
  7. 2026MOSAIC: Masked Outsourcing of Secure AI Computationscited by 1
  8. 2026Hive: A Multi-Agent Infrastructure for Algorithm- and Task-Level Scalingcited by 0
  9. 2026Streaming Communication in Multi-Agent Reasoningcited by 0
  10. 2026KernelFlume: Elastic Core-Attention Scaling for Agentic Long-Context Decodingcited by 0
  11. 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetchingcited by 0
  12. 2026BIO-MEMART: Biometric-Aware KV Cache Memory for Multi-User LLM Agentscited by 0
3 more
  1. 2026Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDscited by 0
  2. 2026Dynamic HBM Repartitioning for Multi-Turn MoE Servingcited by 0
  3. 2026PipeSwift: Revisiting Pipeline Parallelism for Large-Scale Completion-Oriented Agentic LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference
  2. 2026CALVO: Improve Serving Efficiency for LLM Inferences with Intense Network Demands
  3. 2026End-to-End Latency-Minimizing and Load-Balanced Request Scheduling for Edge LLM Inference in Agentic AI Services
  4. 2026From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems
  5. 2026When Does Disaggregation Pay? Simulating Prefill--Decode--Attention--FFN Specialization for Agentic LLM Inference
  6. 2026From Inference Engine to Inference Control Plane: Connecting vLLM, llm-d, and the Evolution of Efficient Distributed LLM Serving
  7. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs
  8. 2026DualDecoder: Accelerate Long Context LLM Inference by Predictive Prefetch