mlsanityData to 7 Oct 2026

GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM

arXiv 2403.05527 ↗First posted 8 Mar 2024cs.LG cs.AI cs.CLcited by 155

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Key-value (KV) caching has become the de-facto to accelerate generation speed for large language models (LLMs) inference. However, the growing cache demand with increasing sequence length has transformed LLM inference to be a memory bound problem, significantly constraining the system throughput. Existing methods rely on dropping unimportant tokens or quantizing all entries uniformly. Such methods, however, often incur high approximation errors to represent the compressed matrices. The autoregressive decoding process further compounds the error of each step, resulting in critical deviation in model generation and deterioration of performance. To tackle this challenge, we propose GEAR, an efficient KV cache compression framework that achieves near-lossless high-ratio compression. GEAR first applies quantization to majority of entries of similar magnitudes to ultra-low precision. It then employs a low rank matrix to approximate the quantization error, and a sparse matrix to remedy individual errors from outlier entries. By adeptly integrating three techniques, GEAR is able to fully exploit their synergistic potentials. Our experiments demonstrate that compared to alternatives, GEAR achieves near-lossless 4-bit KV cache compression with up to 2.38x throughput improvement, while reducing peak-memory size up to 2.29x. Our code is publicly available at https://github.com/HaoKang-Timmy/GEAR.

Key facts. Cited by 155 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 38 of its 44 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

38 of 44 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
  2. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  3. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
23 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  2. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  4. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  7. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  9. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  10. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  11. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  13. 2022Efficiently Scaling Transformer Inferencecited by 529
  14. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  15. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  16. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  17. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  18. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  19. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 234
  20. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  21. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  22. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
  23. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 82

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 46; 2025: 58; 2026: 51.

155 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  3. 2024CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusioncited by 211
  4. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  5. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  6. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  7. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  8. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  9. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  10. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  11. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 94
  12. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
38 more
  1. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  2. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compressioncited by 71
  3. 2024Exploiting LLM Quantizationcited by 61
  4. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 59
  5. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
  6. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  7. 2024QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overheadcited by 47
  8. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  9. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  10. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  11. 2024ZipVL: Efficient Large Vision-Language Models with Dynamic Token Sparsificationcited by 43
  12. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMscited by 42
  13. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 42
  14. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 38
  15. 2025Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architecturescited by 38
  16. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  17. 2024Fast State Restoration in LLM Serving with HCachecited by 32
  18. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residualscited by 32
  19. 2025MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inferencecited by 31
  20. 2024Effectively Compress KV Heads for LLMcited by 30
  21. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 30
  22. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  23. 2024Compute Or Load KV Cache? Why Not Both?cited by 25
  24. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environmentscited by 23
  25. 2024MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantizationcited by 22
  26. 2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
  27. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 21
  28. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  29. 2025Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppressioncited by 21
  30. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
  31. 2025Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMscited by 20
  32. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurationscited by 19
  33. 2024QSpec: Speculative Decoding with Complementary Quantization Schemescited by 18
  34. 2026Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantizationcited by 18
  35. 2024Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionariescited by 17
  36. 2025Online Scheduling for LLM Inference with KV Cache Constraintscited by 17
  37. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregationcited by 16
  38. 2025Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraintscited by 16

50 most cited of 155 All 155 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization
  2. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
  3. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  4. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  5. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  6. 2026MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
  7. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
  8. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction