mlsanityData to 14 Oct 2026

HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs

arXiv 2507.19823 ↗First posted 26 Jul 2025cs.CL cs.AI cs.LGcited by 3

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Processing long-context inputs with large language models presents a significant challenge due to the enormous memory requirements of the Key-Value (KV) cache during inference. Existing KV cache compression methods exhibit noticeable performance degradation when memory is reduced by more than 85%. Additionally, strategies that leverage GPU-CPU collaboration for approximate attention remain underexplored in this setting. We propose HCAttention, a heterogeneous attention computation framework that integrates key quantization, value offloading, and dynamic KV eviction to enable efficient inference under extreme memory constraints. The method is compatible with existing transformer architectures and does not require model fine-tuning. Experimental results on the LongBench benchmark demonstrate that our approach preserves the accuracy of full-attention model while shrinking the KV cache memory footprint to 25% of its original size. Remarkably, it stays competitive with only 12.5% of the cache, setting a new state-of-the-art in LLM KV cache compression. To the best of our knowledge, HCAttention is the first to extend the Llama-3-8B model to process 4 million tokens on a single A100 GPU with 80GB memory.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 45 references that are in the map. First posted on arXiv 26 Jul 2025.

Builds on

39 of 45 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  2. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  3. 2024vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving22 Jul
  4. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  6. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget7 Apr
  7. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference14 Mar
  8. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  9. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2020Longformer: The Long-Document Transformercited by 3,413
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2024Mixtral of Expertscited by 1,823
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
17 more
  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  2. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  4. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
  5. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 206
  6. 2020Fast Transformers with Clustered Attentioncited by 123
  7. 2024Transformers are Multi-State RNNscited by 120
  8. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  9. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 76
  10. 2025Beyond Single-Turn: A Survey on Multi-Turn Interactions with Large Language Modelscited by 62
  11. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelinescited by 49
  12. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  13. 2025A Survey on Multi-Turn Interaction Capabilities of Large Language Modelscited by 30
  14. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurationscited by 19
  15. 2024Beyond KV Caching: Shared Attention for Efficient LLMscited by 10
  16. 2024FocusLLM: Precise Understanding of Long Context by Dynamic Condensingcited by 10
  17. 2024LIFT: Improving Long Context Understanding Through Long Input Fine-Tuningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026KV Cache Offloading for Context-Intensive Taskscited by 1
  2. 2026DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compressioncited by 1
  3. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
  2. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
  3. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  4. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
  5. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  6. 2025ZSMerge: Zero-Shot KV Cache Compression for Memory-Efficient Long-Context LLMs
  7. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  8. 2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharing