mlsanityData to 14 Oct 2026

BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inference

arXiv 2410.23079 ↗First posted 30 Oct 2024cs.CL cs.AIcited by 9

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models (LLMs) are essential in natural language processing but often struggle with inference speed and computational efficiency, limiting real-time deployment. The key-value (KV) cache mechanism reduces computational overhead in transformer models, but challenges in maintaining contextual understanding remain. In this paper, we propose BUZZ, a novel KV caching algorithm that leverages structured contextual information to minimize cache memory usage while enhancing inference speed. BUZZ employs a beehive-structured sparse cache, incorporating a sliding window to capture recent information and dynamically segmenting historical tokens into chunks to prioritize important tokens in local neighborhoods. We evaluate BUZZ on four real-world datasets: CNN/Daily Mail, XSUM, Wikitext, and 10-QA. Our results demonstrate that BUZZ (1) reduces cache memory usage by 2.5× in LLM inference while maintaining over 99% accuracy in long-text summarization, and (2) surpasses state-of-the-art performance in multi-document question answering by 7.69% under the same memory limit, where full cache methods encounter out-of-memory issues. Additionally, BUZZ achieves significant inference speedup with a logn time complexity. The code is available at https://github.com/JunqiZhao888/buzz-llm.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 50 of its 58 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

50 of 58 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Models23 May
  2. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Models17 May
  3. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  4. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM8 Mar
  5. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference14 Feb
  6. 2024SubGen: Token Generation in Sublinear Time and Memory8 Feb
  7. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  8. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  9. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2023Mistral 7Bcited by 3,164
  11. 2016Pointer Sentinel Mixture Modelscited by 3,096
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,916
29 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,512
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  9. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  10. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  11. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  12. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  13. 2021Transformer in Transformercited by 633
  14. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  15. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  16. 2022Post-training Quantization on Diffusion Modelscited by 267
  17. 2022Neighborhood Attention Transformercited by 239
  18. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  19. 2023Using an LLM to Help With Code Understandingcited by 228
  20. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 162
  21. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  22. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 113
  23. 2023HyperAttention: Long-context Attention in Near-Linear Timecited by 85
  24. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains Morecited by 69
  25. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  26. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 44
  27. 2023DilateFormer: Multi-Scale Dilated Transformer for Visual Recognitioncited by 34
  28. 2023New Subset Selection Algorithms for Low Rank Approximation: Offline and Onlinecited by 11
  29. 2024Faster Algorithms for Schatten-p Low Rank Approximationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 4.

9 papers.

  1. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  3. 2025Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inferencecited by 18
  4. 2026Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigationcited by 11
  5. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 6
  6. 2025DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Accelerationcited by 4
  7. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  8. 2026SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXLcited by 2
  9. 2026Dynamic Thinking-Token Selection for Efficient Reasoning in Large Reasoning Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  2. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  3. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  4. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
  5. 2026Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM
  6. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
  7. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
  8. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection