mlsanityData to 7 Oct 2026

KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inference

arXiv 2511.11907 ↗First posted 14 Nov 2025cs.DC cs.AIcited by 5

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Language models (LMs) underpin emerging mobile and embedded AI applications like meeting and video summarization and document analysis, which often require processing multiple long-context inputs. Running an LM locally on-device improves privacy, enables offline use, and reduces cost, but long-context inference quickly hits a memory capacity wall as the key-value (KV) cache grows linearly with context length and batch size. Existing KV-cache offloading schemes are designed to transfer cache data from GPU memory to CPU memory; however, they are not suitable for embedded and mobile systems, where the CPU and GPU (or NPU) typically share a unified memory and the non-volatile secondary storage (disk) offers limited I/O bandwidth. We present KVSwap, a software framework tailored for local devices that achieves high memory efficiency while effectively leveraging disk storage. KVSwap stores the full cache on disk, uses highly compact in-memory metadata to predict which entries to preload, overlaps computation with hardware-aware disk access, and orchestrates read patterns to match storage device characteristics. Our evaluation shows that across representative LMs and storage types, KVSwap delivers higher throughput under tight memory budgets while maintaining generation quality over existing KV cache offloading schemes.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 48 of its 83 references that are in the map. First posted on arXiv 14 Nov 2025.

Builds on

48 of 83 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs20 Mar
  2. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection5 Nov
  3. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  6. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion26 May
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  8. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2025Qwen2.5-VL Technical Reportcited by 5,418
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  9. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  11. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  12. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
26 more
  1. 2020Reformer: The Efficient Transformercited by 1,658
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  6. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  7. 2023A Survey on Multimodal Large Language Modelscited by 898
  8. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  10. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  11. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  12. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  13. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  14. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  15. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  16. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  17. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  18. 2022Language model compression with weighted low-rank factorizationcited by 192
  19. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  20. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  21. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
  22. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 87
  23. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 74
  24. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  25. 2021Sparse Attention with Linear Unitscited by 41
  26. 2023Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspectivecited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Servingcited by 9
  2. 2026Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devicescited by 8
  3. 2026DUAL-BLADE: Dual-Path NVMe-Direct KV-Cache Offloading for Edge LLM Inferencecited by 2
  4. 2026From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMscited by 0
  5. 2026SPLASH: Co-Designing Sparse Attention with High-Bandwidth Flash for Efficient Long-Context Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  2. 2026KV Cache Offloading for Context-Intensive Tasks
  3. 2026TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Caching
  4. 2026HillInfer: Efficient Long-Context LLM Inference on the Edge with Hierarchical KV Eviction using SmartSSD
  5. 2026SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference
  6. 2026ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference
  7. 2025A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs
  8. 2026TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference