mlsanityData to 7 Oct 2026

Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs

arXiv 2609.34727 ↗First posted 28 Sep 2026cs.OS cs.AI cs.DBcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

On-device large language model (LLM) serving is a cornerstone of local-first personal intelligence, offering users data sovereignty, strong privacy guarantees, and freedom from cloud API latency and cost. Although KV caching is widely used to reduce latency in long-context inference, existing designs were primarily optimized for cloud GPUs with dynamic execution environments and abundant memory bandwidth. These architectural assumptions do not hold on mobile NPUs, where computation graphs must be statically compiled and both memory capacity and I/O bandwidth are severely constrained. In this work, we present a compute-storage co-design for mobile-centric prefix and non-prefix KV reuse. We first propose an intra-graph mechanism that maps selective KV recomputation onto static NPU graphs, reconciling algorithmic dynamicity with NPU staticity. We further develop an inter-graph scheduler to optimize chunk merging and minimize padding with dynamic programming. To address mobile bandwidth limitations, we introduce a hierarchical KV manager featuring a tree-hash-semantic hybrid structure, along with cost-aware prefetching and eviction policies. We also build a two-dimensional pipeline that overlaps KV loading, rerotation, and storage with NPU execution, hiding data-movement latency. Experiments across representative on-device workloads and LLMs show that our design reduces time-to-first-token (TTFT) by 40-60% compared with no reuse and prefix-only caching.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 48 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

27 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices12 Jun
  2. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  3. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory12 Dec

Other references · 24

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  6. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 800
  7. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  8. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 255
  9. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  10. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  11. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
  12. 2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 81
12 more
  1. 2024Fast On-device LLM Inference with NPUscited by 58
  2. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 47
  3. 2025Strata: Hierarchical Context Caching for Long Context Language Model Servingcited by 25
  4. 2025Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inferencecited by 20
  5. 2024SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobilecited by 12
  6. 2025MobileRAG: A Fast, Memory-Efficient, and Energy-Efficient Method for On-Device RAGcited by 9
  7. 2025Scaling LLM Test-Time Compute with Mobile NPU on Smartphonescited by 9
  8. 2025MicroNN: An On-device Disk-resident Updatable Vector Databasecited by 6
  9. 2024Neuralink: Fast LLM Inference on Smartphones with Neuron Co-Activation Linkingcited by 4
  10. 2025T-MAN: Enabling End-to-End Low-Bit LLM Inference on NPUs via Unified Table Lookupcited by 4
  11. 2026ClawMobile: Rethinking Smartphone-Native Agentic Systemscited by 2
  12. 2026SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inferencecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
  2. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
  3. 2026Adaptive KV Cache Reuse for Fast Long-Context LLM Serving
  4. 2026Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service
  5. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse
  6. 2025Scalable Processing-Near-Memory for 1M-Token LLM Inference: CXL-Enabled KV-Cache Management Beyond GPU Limits
  7. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  8. 2025PRESERVE: Prefetching Model Weights and KV-Cache in Distributed LLM Serving