mlsanityData to 14 Oct 2026

CTkvr: KV Cache Retrieval for Long-Context LLMs via Centroid then Token Indexing

arXiv 2512.15550 ↗First posted 17 Dec 2025cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models (LLMs) are increasingly applied in long-context scenarios such as multi-turn conversations. However, long contexts pose significant challenges for inference efficiency, including high memory overhead from Key-Value (KV) cache and increased latency due to excessive memory accesses. Recent methods for dynamic KV selection struggle with trade-offs: block-level indexing degrades accuracy by retrieving irrelevant KV entries, while token-level indexing incurs high latency from inefficient retrieval mechanisms. In this paper, we propose CTKVR, a novel centroid-then-token KV retrieval scheme that addresses these limitations. CTKVR leverages a key observation: query vectors adjacent in position exhibit high similarity after Rotary Position Embedding (RoPE) and share most of their top-k KV cache entries. Based on this insight, CTKVR employs a two-stage retrieval strategy: lightweight centroids are precomputed during prefilling for centroid-grained indexing, followed by token-level refinement for precise KV retrieval. This approach balances retrieval efficiency and accuracy. To further enhance performance, we implement an optimized system for indexing construction and search using CPU-GPU co-execution. Experimentally, CTKVR achieves superior performance across multiple benchmarks with less than 1% accuracy degradation. Meanwhile, CTKVR delivers 3 times and 4 times throughput speedups on Llama-3-8B and Yi-9B at 96K context length across diverse GPU hardware.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 30 references that are in the map. First posted on arXiv 17 Dec 2025.

Builds on

17 of 30 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection5 Nov
  2. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  3. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  4. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 11

  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  6. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 144
  7. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  8. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  9. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  10. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  11. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clustering
  2. 2025CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation
  3. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression
  4. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
  5. 2026ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMs
  6. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods
  7. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference
  8. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference