mlsanityData to 7 Oct 2026

DynSplit-KV: Dynamic Semantic Splitting for KVCache Compression in Efficient Long-Context LLM Inference

arXiv 2602.03184 ↗First posted 3 Feb 2026cs.LG cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Although Key-Value (KV) Cache is essential for efficient large language models (LLMs) inference, its growing memory footprint in long-context scenarios poses a significant bottleneck, making KVCache compression crucial. Current compression methods rely on rigid splitting strategies, such as fixed intervals or pre-defined delimiters. We observe that rigid splitting suffers from significant accuracy degradation (ranging from 5.5% to 55.1%) across different scenarios, owing to the scenario-dependent nature of the semantic boundaries. This highlights the necessity of dynamic semantic splitting to match semantics. To achieve this, we face two challenges. (1) Improper delimiter selection misaligns semantics with the KVCache, resulting in 28.6% accuracy loss. (2) Variable-length blocks after splitting introduce over 73.1% additional inference overhead. To address the above challenges, we propose DynSplit-KV, a KVCache compression method that dynamically identifies delimiters for splitting. We propose: (1) a dynamic importance-aware delimiter selection strategy, improving accuracy by 49.9%. (2) A uniform mapping strategy that transforms variable-length semantic blocks into a fixed-length format, reducing inference overhead by 4.9x. Experiments show that DynSplit-KV achieves the highest accuracy, 2.2x speedup compared with FlashAttention and 2.6x peak memory reduction in long-context scenarios.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 26 references that are in the map. First posted on arXiv 3 Feb 2026.

Builds on

18 of 26 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025SABlock: Semantic-Aware KV Cache Eviction with Adaptive Compression Block Size26 Oct
  2. 2025SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching1 Apr
  3. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference1 Feb
  4. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 11

  1. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  7. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  9. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  10. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  11. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
  2. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  3. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  4. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  5. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
  6. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  7. 2026MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
  8. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models