mlsanityData to 7 Oct 2026

vToken: Token-Level Virtualization for Reclaimable KV Caches

arXiv 2608.13263 ↗First posted 13 Aug 2026cs.AI cs.DC cs.OScited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language model serving faces a critical memory bottleneck: the KV cache grows with sequence length and batch size. PagedAttention uses fixed-size memory blocks to reduce allocator-level fragmentation, but recent KV eviction algorithms operate at a token granularity finer than block-level management. This mismatch causes intra-block fragmentation, leaving a large fraction of allocated KV memory unreclaimable. We present vToken, a lightweight token-level virtualization layer that decouples logical token liveness from physical block placement. vToken maintains a stable logical token view through token-table indirection and realizes physical reclamation by repacking live tokens asynchronously. The design preserves PagedAttention kernels and CUDA Graph compatibility. We implement vToken in vLLM and evaluate it with H2O, Random, and Scissorhands across models. Compared with a paired Naive-Evict baseline, vToken reduces retained KV blocks per request by 27.2%--72.3% and improves SLA-constrained throughput by up to 1.37×. Under a constrained active-KV budget, it extends the maximum feasible concurrency by up to 2×, while reducing the per-policy integration footprint from 500+ lines to under 50.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 35 references that are in the map. First posted on arXiv 13 Aug 2026.

Builds on

21 of 35 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Strata: Hierarchical Context Caching for Long Context Language Model Serving26 Aug
  2. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction4 Dec
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  4. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  5. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  7. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  5. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  11. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  12. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 93
2 more
  1. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  2. 2026Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttentioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SparseEngine: Sparse-First Inference Enginecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
  2. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
  3. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse
  4. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  5. 2026Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs
  6. 2026ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Management
  7. 2025GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction
  8. 2026Sub-Token Routing for KV Cache Compression