mlsanityData to 14 Oct 2026

CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration

arXiv 2608.30295 ↗First posted 31 Aug 2026cs.LG cs.AIcited by 1

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models (LLMs) have demonstrated strong capabilities in handling long-context tasks, but processing such long contexts remains challenging due to the substantial memory requirements and inference latency. In this work, we discover that certain attention heads exhibit sequential consistency in their attention patterns, which can be persistently identified using a coefficient-of-variation-based algorithm. Inspired by this observation, we propose CateKV, a hybrid KV cache method that retains only critical token information for consistent heads, thereby reducing KV cache size and computational overhead, while preserving the majority of KV pairs in adaptive heads to ensure high accuracy. We show the unique characteristics of our algorithm and its extension with existing acceleration methods. Comprehensive evaluations on long-context benchmarks show that, while maintaining accuracy comparable to full attention, CateKV reduces memory usage by up to 2.72× and accelerates decoding by 2.18× in single-sample inputs, and boosts throughput by 3.96× in batch scenarios.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 40 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

33 of 40 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  2. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  3. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  4. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  5. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  6. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  8. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  9. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference21 May
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  11. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
1 more
  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 20

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2023Qwen Technical Reportcited by 3,344
  5. 2016Pointer Sentinel Mixture Modelscited by 3,096
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  11. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  12. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
8 more
  1. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  2. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  3. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  4. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  5. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
  6. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  7. 2024ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Cachingcited by 48
  8. 2023Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answeringcited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Understanding and Exploiting Diagonal Attention Sparsity in Autoregressive Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs
  2. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  3. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  4. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  5. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
  6. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
  7. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  8. 2025Training-free Context-adaptive Attention for Efficient Long Context Modeling