mlsanityData to 14 Oct 2026

SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching

arXiv 2504.00970 ↗First posted 1 Apr 2025cs.CL cs.AI cs.LGcited by 10

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models face significant computational and memory challenges when processing long contexts. During inference, efficient management of the key-value (KV) cache, which stores intermediate activations for autoregressive generation, is critical to reducing memory overhead and improving computational efficiency. Traditional token-level efficient KV caching methods overlook semantic information, treating tokens independently without considering their semantic relationships. Meanwhile, existing semantic-preserving KV cache management approaches often suffer from substantial memory usage and high time-to-first-token. To address these limitations, we propose SentenceKV, a novel sentence-level semantic KV caching approach designed to enhance inference efficiency while preserving semantic coherence. During prefilling, SentenceKV groups tokens based on sentence-level semantic similarity, compressing sentence representations into concise semantic vectors stored directly on the GPU, while individual KV pairs are offloaded to CPU. During decoding, SentenceKV generates tokens by selectively retrieving semantically relevant sentence-level KV entries, leveraging the semantic similarity between the prefilling-stage semantic vectors and decoding-stage queries. This ensures efficient and contextually accurate predictions, minimizing the loading of redundant or irrelevant data into GPU memory and significantly reducing memory overhead while maintaining stable inference latency, even for extremely long contexts. Extensive evaluations on benchmarks including PG-19, LongBench, and Needle-In-A-Haystack demonstrate that SentenceKV significantly outperforms state-of-the-art methods in both efficiency and memory usage, without compromising model accuracy.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 36 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

28 of 36 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference1 Feb
  2. 2025Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Heads25 Jan
  3. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs19 Dec
  4. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  5. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  6. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption25 Jul
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  8. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  9. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  11. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  3. 2023A Survey of Large Language Modelscited by 2,576
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  9. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  10. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  11. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  12. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
4 more
  1. 2024Transformers are Multi-State RNNscited by 120
  2. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 57
  3. 2024SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separatorcited by 41
  4. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 8.

10 papers.

  1. 2026From Similarity to Vulnerability: Key Collision Attack on LLM Semantic Cachingcited by 4
  2. 2025Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIMcited by 3
  3. 2025Supporting Dynamic Agentic Workloads: How Data and Agents Interactcited by 2
  4. 2026Rethinking Token Prediction: Tree-Structured Diffusion Language Modelcited by 1
  5. 2026LaCache: Robust Semantic Caching for LLM Servingcited by 1
  6. 2026DynSplit-KV: Dynamic Semantic Splitting for KVCache Compression in Efficient Long-Context LLM Inferencecited by 0
  7. 2026LLM-CoOpt: A Co-Design and Optimization Framework for Efficient LLM Inference on Heterogeneous Platformscited by 0
  8. 2026ZoomR: Memory Efficient Reasoning through Multi-Granularity Key Value Retrievalcited by 0
  9. 2026SALT: Salience-Aware Lexical Trie for Long-Context Compressioncited by 0
  10. 2026CineMem: Attention-Aligned Fine-grained Memory Retrieval for Multi-Shot Long Video Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
  2. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
  3. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
  4. 2025WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference
  5. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
  6. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
  7. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  8. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference