mlsanityData to 14 Oct 2026

SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference

arXiv 2604.21231 ↗First posted 23 Apr 2026cs.NI cs.AI cs.PFcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Efficient inference for on-device Large Language Models (LLMs) remains challenging due to limited hardware resources and the high cost of the prefill stage, which processes the full input context to construct Key-Value (KV) caches. We present SparKV, an adaptive KV loading framework that combines cloud-based KV streaming with on-device computation. SparKV models the cost of individual KV chunks and decides whether each chunk should be streamed or computed locally, while overlapping the two execution paths to reduce latency. To handle fluctuations in wireless connectivity and edge resource availability, SparKV further refines offline-generated schedules at runtime to rebalance communication and computation costs. Experiments across diverse datasets, LLMs, and edge devices show that SparKV reduces Time-to-First-Token by 1.3x-5.1x with negligible impact on response quality, while lowering per-request energy consumption by 1.5x to 3.3x, demonstrating its robustness and practicality for real-world on-device deployment.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 58 references that are in the map. First posted on arXiv 23 Apr 2026.

Builds on

31 of 58 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy4 Oct
  2. 2024Compute Or Load KV Cache? Why Not Both?4 Oct
  3. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  4. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  5. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2023Mistral 7Bcited by 3,164
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  11. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
12 more
  1. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  3. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  4. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  5. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  6. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  7. 2021Efficient Attentions for Long Document Summarizationcited by 345
  8. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 137
  9. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
  10. 2024SageAttention2: Efficient Attention with Thorough Outlier Smoothing and Per-thread INT4 Quantizationcited by 114
  11. 2024Fast On-device LLM Inference with NPUscited by 62
  12. 2024EdgeShard: Efficient LLM Inference via Collaborative Edge Computingcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching
  2. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  3. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  4. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  5. 2026LeanStream: A Speculate-and-Refine Streaming Framework for Efficient on-Device LLM Inference
  6. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  7. 2026TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference
  8. 2026When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference