# DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference

arXiv: https://arxiv.org/abs/2602.21548 · HTML: https://mlsanity.com/paper/2602.21548/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 24 of its 50 references that are in the map.
- First posted on arXiv 25 Feb 2026.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention](https://mlsanity.com/paper/2403.19708/) (2024)
- [Evaluation and Benchmarking of LLM Agents: A Survey](https://mlsanity.com/paper/2507.21504/) (2025)
- [Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures](https://mlsanity.com/paper/2505.09343/) (2025)
- [Fast State Restoration in LLM Serving with HCache](https://mlsanity.com/paper/2410.05004/) (2024)
- [From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Users](https://mlsanity.com/paper/2508.17281/) (2025)
- [PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications](https://mlsanity.com/paper/2505.07203/) (2025)
- [CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control](https://mlsanity.com/paper/2601.22705/) (2026)
- [KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputation](https://mlsanity.com/paper/2411.17089/) (2024)
- [TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimization](https://mlsanity.com/paper/2505.19586/) (2025)
- [TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving](https://mlsanity.com/paper/2508.17219/) (2025)

## Built on by (most cited first, 15 of 15)

- [TraceLab: Characterizing Coding Agent Workloads for LLM Serving](https://mlsanity.com/paper/2606.30560/) (2026)
- [Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving](https://mlsanity.com/paper/2604.25899/) (2026)
- [Efficient Serving for Dynamic Agent Workflows with Prediction-based KV-Cache Management](https://mlsanity.com/paper/2605.06472/) (2026)
- [A Token/KV-Cache Communication Media Selection and Resource Allocation Strategy for Multi-Agent Collaboration](https://mlsanity.com/paper/2605.25422/) (2026)
- [Stage-Aware Communication Scheduling for Disaggregated LLM Serving](https://mlsanity.com/paper/2603.17456/) (2026)
- [TENT: A Declarative Slice Spraying Engine for Performant and Resilient Data Movement in Disaggregated LLM Serving](https://mlsanity.com/paper/2604.00368/) (2026)
- [MOSAIC: Masked Outsourcing of Secure AI Computations](https://mlsanity.com/paper/2607.29221/) (2026)
- [Hive: A Multi-Agent Infrastructure for Algorithm- and Task-Level Scaling](https://mlsanity.com/paper/2604.17353/) (2026)
- [Streaming Communication in Multi-Agent Reasoning](https://mlsanity.com/paper/2606.05158/) (2026)
- [KernelFlume: Elastic Core-Attention Scaling for Agentic Long-Context Decoding](https://mlsanity.com/paper/2606.29207/) (2026)
- [OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching](https://mlsanity.com/paper/2608.08097/) (2026)
- [BIO-MEMART: Biometric-Aware KV Cache Memory for Multi-User LLM Agents](https://mlsanity.com/paper/2609.08566/) (2026)
- [Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs](https://mlsanity.com/paper/2609.11744/) (2026)
- [Dynamic HBM Repartitioning for Multi-Turn MoE Serving](https://mlsanity.com/paper/2609.13537/) (2026)
- [PipeSwift: Revisiting Pipeline Parallelism for Large-Scale Completion-Oriented Agentic LLM Serving](https://mlsanity.com/paper/2609.16491/) (2026)

## Abstract (arXiv)

The performance of multi-turn, agentic LLM inference is increasingly dominated by KV-Cache storage I/O rather than computation. In prevalent disaggregated architectures, loading the massive KV-Cache from external storage creates a fundamental imbalance: storage NICs on prefill engines become bandwidth-saturated, while those on decoding engines remain idle. This asymmetry severely constrains overall system throughput. We present DualPath, an inference system that breaks this bottleneck by introducing dual-path KV-Cache loading. Beyond the traditional storage-to-prefill path, DualPath enables a novel storage-to-decode path, in which the KV-Cache is loaded into decoding engines and then efficiently transferred to prefill engines via RDMA over the compute network. DualPath combines this optimized data path -- which inherently avoids network congestion and avoids interference with latency-critical model execution communications -- with a global scheduler that dynamically balances load across prefill and decode engines. Our evaluation on three models with production agentic workloads demonstrates that DualPath improves offline inference throughput by up to 1.87× on our in-house inference system. It can also improve online serving throughput by an average factor of 1.96× without violating SLO.