# SABlock: Semantic-Aware KV Cache Eviction with Adaptive Compression Block Size

arXiv: https://arxiv.org/abs/2510.22556 · HTML: https://mlsanity.com/paper/2510.22556/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- First posted on arXiv 26 Oct 2025.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Built on by (most cited first, 3 of 3)

- [Adaptive Mass-Segmented KV Compression for Long-Context Reasoning](https://mlsanity.com/paper/2605.23200/) (2026)
- [From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs](https://mlsanity.com/paper/2606.09508/) (2026)
- [DynSplit-KV: Dynamic Semantic Splitting for KVCache Compression in Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2602.03184/) (2026)

## Abstract (arXiv)

The growing memory footprint of the Key-Value (KV) cache poses a severe scalability bottleneck for long-context Large Language Model (LLM) inference. While KV cache eviction has emerged as an effective solution by discarding less critical tokens, existing token-, block-, and sentence-level compression methods struggle to balance semantic coherence and memory efficiency. To this end, we introduce SABlock, a underlinesemantic-aware KV cache eviction framework with underlineadaptive underlineblock sizes. Specifically, SABlock first performs semantic segmentation to align compression boundaries with linguistic structures, then applies segment-guided token scoring to refine token importance estimation. Finally, for each segment, a budget-driven search strategy adaptively determines the optimal block size that preserves semantic integrity while improving compression efficiency under a given cache budget. Extensive experiments on long-context benchmarks demonstrate that SABlock consistently outperforms state-of-the-art baselines under the same memory budgets. For instance, on Needle-in-a-Haystack (NIAH), SABlock achieves 99.9% retrieval accuracy with only 96 KV entries, nearly matching the performance of the full-cache baseline that retains up to 8K entries. Under a fixed cache budget of 1,024, SABlock further reduces peak memory usage by 46.28% and achieves up to 9.5x faster decoding on a 128K context length.