mlsanityData to 7 Oct 2026

IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs

arXiv 2604.10539 ↗First posted 12 Apr 2026cs.LG cs.AIcited by 4

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Key-Value (KV) cache plays a crucial role in accelerating inference in large language models (LLMs) by storing intermediate attention states and avoiding redundant computation during autoregressive generation. However, its memory footprint scales linearly with sequence length, often leading to severe memory bottlenecks on resource-constrained hardware. Prior work has explored offloading KV cache to the CPU while retaining only a subset on the GPU, but these approaches often rely on imprecise token selection and suffer performance degradation in long-generation tasks such as chain-of-thought reasoning. In this paper, we propose a novel KV cache management strategy, IceCache, which integrates semantic token clustering with PagedAttention. By organizing semantically related tokens into contiguous memory regions managed by a hierarchical, dynamically updatable data structure, our method enables more efficient token selection and better utilization of memory bandwidth during CPU-GPU transfers. Experimental results on LongBench show that, with a 256-token budget, IceCache maintains 99% of the original accuracy achieved by the full KV cache model. Moreover, compared to other offloading-based methods, IceCache attains competitive or even superior latency and accuracy while using only 25% of the KV cache token budget, demonstrating its effectiveness in long-sequence scenarios. The code is available on our project website at https://yuzhenmao.github.io/IceCache/.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 27 references that are in the map. First posted on arXiv 12 Apr 2026.

Builds on

25 of 27 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  2. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  3. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  4. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  5. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2020Reformer: The Efficient Transformercited by 1,658
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  9. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  10. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  11. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  12. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
6 more
  1. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  2. 2021Memory-efficient Transformers via Top-k Attentioncited by 59
  3. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  4. 2024IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUscited by 8
  5. 2015Fast k-Nearest Neighbour Search via Dynamic Continuous Indexingcited by 6
  6. 2017Fast k-Nearest Neighbour Search via Prioritized DCIcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compressioncited by 2
  2. 2026AVQ-Attention: Adaptive Vector-Quantized Attentioncited by 1
  3. 2026H2MT: Semantic Hierarchy-Aware Hierarchical Memory Transformercited by 0
  4. 2026SPLASH: Co-Designing Sparse Attention with High-Bandwidth Flash for Efficient Long-Context Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Management
  4. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  5. 2026Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
  6. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  7. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
  8. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models