mlsanityData to 14 Oct 2026

AnchorKV: Anchor-Residual KV Cache Compression

arXiv 2608.02901 ↗First posted 3 Aug 2026cs.LG cs.CLcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

The key-value (KV) cache is the primary memory bottleneck in long-context LLM inference. Existing approaches attack it from opposite ends: eviction methods permanently discard tokens, degrading performance whenever a discarded token later proves essential, while quantization methods retain all tokens at low precision but offer limited compression. We propose AnchorKV, a compression scheme that shrinks the cache by 20× without discarding a single token. AnchorKV represents the cache using a small set of anchors stored exactly, expresses every other token through its most similar anchor, and refines only those whose approximation most affects the model's output. AnchorKV consistently preserves accuracy across models and datasets, retaining 99% of the full-cache score at the 70B scale, while keeping the entire context at a fraction of its cost.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 27 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

16 of 27 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inference9 Oct
  2. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution1 Oct
  3. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective6 Feb
  4. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  5. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks11 Jul
  6. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar

Other references · 10

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  6. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  7. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  8. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 125
  9. 2025CommVQ: Commutative Vector Quantization for KV Cache Compressioncited by 28
  10. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correctioncited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026PAGE: Partition-Aware Gated KV-Cache Evictioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AnchorKV: Safety-Aware KV Cache Compression via Soft Penalty with a Refusal Anchor
  2. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
  3. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
  4. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  5. 2026KVSculpt: KV Cache Compression as Distillation
  6. 2026MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
  7. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
  8. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference