mlsanityData to 7 Oct 2026

SlimKV: Joint Token-Feature KV Cache Compression with Reconstruction-Free Beacon Attention

arXiv 2610.02953 ↗First posted 2 Oct 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context LLM serving is increasingly bottlenecked by KV-cache memory, especially in resource-constrained scenarios. Among existing KV-cache compression strategies, token-wise methods reduce cached states but risk information loss through eviction or condensation, while feature-wise methods reduce per-token KV dimensions but can require full-dimensional reconstruction to apply positional embedding, limiting decoding speedups. We introduce SlimKV, a question-agnostic joint token-feature KV-cache compression method. SlimKV uses low-rank-aware training to compress long contexts into beacon memory states with latent KV representations, together with layer-adaptive rank allocation. We further uncover a positional asymmetry: removing key-side RoPE affects beacon and raw tokens differently, with much smaller degradation for beacon tokens. Exploiting this asymmetry, SlimKV trains beacon KV projections under a K-RoPE-free constraint and enables latent-space attention during decoding, mitigating reconstruction latency. On LongBench, SlimKV outperforms baselines at 16x/32x compression and remains leading at 4x/8x, where it retains over 96% of the uncompressed model's score. Needle-in-a-Haystack confirms robustness across evidence positions, and efficiency evaluation shows up to 7.34x attention speedup and 3.38x end-to-end decoding speedup over the uncompressed model at 128K length.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 32 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

24 of 32 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction29 May
  2. 2025Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs20 Feb
  3. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  4. 2024Palu: Compressing KV-Cache with Low-Rank Projection30 Jul
  5. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 17

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  11. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  12. 2023Learning to Compress Prompts with Gist Tokenscited by 379
5 more
  1. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  2. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  3. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  4. 2025A Comprehensive Survey on Long Context Language Modelingcited by 113
  5. 2024Long Context Compression with Activation Beaconcited by 101

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SALS: Sparse Attention in Latent Space for KV cache Compression
  2. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
  3. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  4. 2025EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projection
  5. 2026ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs
  6. 2026KVSculpt: KV Cache Compression as Distillation
  7. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  8. 2025LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Important