mlsanityData to 14 Oct 2026

Sub-Token Routing for KV Cache Compression

arXiv 2604.21335 ↗First posted 23 Apr 2026cs.LG cs.CLcited by 3

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Transformer inference often requires a large KV cache, especially for long-context language modeling and multimodal generation. Existing compression methods usually reduce cache cost by selecting, evicting, quantizing, or compressing cached tokens, or by reducing the visual-token sequence before language-model inference. We introduce sub-token routing, a KV-compression method that adds a finer control axis inside retained tokens. It splits each retained value vector into groups and keeps only selected groups, while leaving query and key states unchanged. The method is designed to work after token-level reduction. First, a token-reduction method determines which tokens are retained. Then, sub-token routing compresses the value states inside those retained tokens. Experiments under matched KV budgets show that adding sub-token routing improves token-level reduction performance in both LLM and VLM settings, including Quest on LLaMA-2-7B and Qwen2.5-7B, and FastV/VisionZip across LLaVA and Qwen-VL models. The gains are larger at smaller KV budgets, suggesting that value-group routing is especially useful when further token removal becomes costly. Overall, token-level reduction and sub-token routing provide complementary ways to reduce KV cost.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 25 references that are in the map. First posted on arXiv 23 Apr 2026.

Builds on

24 of 25 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  2. 2025MadaKV: Adaptive Modality-Perception KV Cache Eviction for Efficient Multimodal Long-Context Inference6 Jun
  3. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  4. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  5. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct

Other references · 19

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2019Towards VQA Models That Can Readcited by 1,718
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  11. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  12. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 402
7 more
  1. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 391
  2. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 313
  3. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 172
  4. 2024Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Modelscited by 126
  5. 2024Efficient Large Multi-modal Models via Visual Context Compressioncited by 46
  6. 2024Residual vector quantization for KV cache compression in large language modelcited by 7
  7. 2026QMoP: Query Guided Mixture-of-Projector for Efficient Visual Token Compressioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compressioncited by 1
  2. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inferencecited by 0
  3. 2026Role-Conditioned Sub-Token Routing for Efficient Vision-Language-Action Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression
  2. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  3. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compression
  4. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  5. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
  6. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
  7. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  8. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference