mlsanityData to 14 Oct 2026

FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Management

arXiv 2511.00868 ↗First posted 2 Nov 2025cs.LGcited by 5

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Model (LLM) serving is increasingly constrained by the growing size of the key-value (KV) cache, which scales with both context length and generation length. Prior work shows that attention is dominated by a small subset of critical tokens, yet existing systems struggle to exploit this efficiently without degrading accuracy, especially in long generation. We make a key observation: the temporal stability of these critical tokens varies significantly across KV heads: some heads consistently focus on the same tokens, while others shift frequently. Building on this insight, we introduce FlexiCache, a hierarchical KV-cache management system that leverages the temporal stability of KV heads to reduce GPU memory usage and computation overhead, while preserving model accuracy. FlexiCache classifies KV heads as stable or unstable: it retains all KV-cache pages from unstable heads in GPU memory, whereas for stable heads, it keeps only the top-K pages on the GPU and offloads the rest to host memory. By exploiting temporal stability, FlexiCache performs periodic reranking for stable heads to fetch newly promoted top pages. Implemented atop vLLM, FlexiCache reduces GPU memory footprint for long-context requests by up to 70%, improves offline serving throughput by 1.38-1.55x, and lowers online token latency by 1.6-2.1x, all while maintaining accuracy in long-context, long-generation scenarios.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 28 references that are in the map. First posted on arXiv 2 Nov 2025.

Builds on

27 of 28 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference8 Oct
  2. 2025Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs2 Mar
  3. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention20 Feb
  4. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  7. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2024Mixtral of Expertscited by 1,823
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  10. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  11. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  12. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
7 more
  1. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  2. 2025Qwen2.5-1M Technical Reportcited by 164
  3. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 144
  4. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 98
  5. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  6. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  7. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026An Efficient Hybrid Sparse Attention with CPU-GPU Parallelism for Long-Context Inferencecited by 3
  2. 2026SPHERICAL KV: Angle-Domain Attention and Rate-Distortion Retention for Efficient Long-Context Inferencecited by 2
  3. 2026Predict, Reuse, and Repair: Accelerating Dynamic Sparse Attention for Long-Context LLM Decodingcited by 2
  4. 2026Move the Query, Not the Cache: Characterizing Cross-Instance Latent Attention Redistribution Across GPU Fabricscited by 1
  5. 2026WakeKV: Reactive, Reversible KV Residency for Heads That Change Their Mindscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
  2. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction
  3. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences
  4. 2025Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference
  5. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
  6. 2026Minima-KV: Retention-Preserving KV Cache Compression with Mixed-Format Paged Attention
  7. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
  8. 2026PersistentKV: Page-Aware Decode Scheduling for Long-Context LLM Serving on Commodity GPUs