mlsanityData to 14 Oct 2026

DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inference

arXiv 2604.26557 ↗First posted 29 Apr 2026cs.DC cs.AI cs.PFcited by 2

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The increasing deployment of Large Language Model (LLM) inference on edge AI systems demands efficient execution under tight memory budgets. A key challenge arises from Key-Value (KV) caches, which often exceed available device memory. Although NVMe-based offloading offers scalable capacity, existing file-based designs rely heavily on the kernel page cache, leading to cache thrashing, unpredictable latency, and high software overhead under memory pressure. We present DUAL-BLADE, a dual-path KV residency framework that dynamically assigns KV tensors to either a page-cache path or an NVMe-direct path based on runtime memory availability. The NVMe-direct path bypasses the filesystem by mapping KV tensors to contiguous logical block address (LBA) regions, enabling low-overhead direct storage access. DUAL-BLADE further incorporates adaptive pipeline parallelism to overlap storage I/O with GPU DMA, improving inference throughput. Our evaluation shows that DUAL-BLADE substantially mitigates I/O bottlenecks, reducing prefill and decode latency by up to 33.1% and 42.4%, respectively, while improving SSD utilization by 2.2x across diverse memory budgets.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 45 references that are in the map. First posted on arXiv 29 Apr 2026.

Builds on

24 of 45 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inference14 Nov
  2. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  3. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference8 Sep
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  5. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2023Mistral 7Bcited by 3,164
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  9. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  10. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  11. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  12. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
6 more
  1. 2022Can Foundation Models Wrangle Your Data?cited by 175
  2. 2024A Review on Edge Large Language Models: Design, Execution, and Applicationscited by 100
  3. 2024LongReward: Improving Long-context Large Language Models with AI Feedbackcited by 34
  4. 2024LogParser-LLM: Advancing Efficient Log Parsing with Large Language Modelscited by 20
  5. 2025A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Servicescited by 17
  6. 2022GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecturecited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026When Cloud Agents Meet Device Agents: Lessons from Hybrid Multi-Agent Systemscited by 1
  2. 2026Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
  2. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  3. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  4. 2026DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference
  5. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  6. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  7. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  8. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading