mlsanityData to 7 Oct 2026

KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Models

arXiv 2409.11057 ↗First posted 17 Sep 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

The bottleneck associated with the key-value(KV) cache presents a significant challenge during the inference processes of large language models. While depth pruning accelerates inference, it requires extensive recovery training, which can take up to two weeks. On the other hand, width pruning retains much of the performance but offers slight speed gains. To tackle these challenges, we propose KVPruner to improve model efficiency while maintaining performance. Our method uses global perplexity-based analysis to determine the importance ratio for each block and provides multiple strategies to prune non-essential KV channels within blocks. Compared to the original model, KVPruner reduces runtime memory usage by 50% and boosts throughput by over 35%. Additionally, our method requires only two hours of LoRA fine-tuning on small datasets to recover most of the performance.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 29 of its 35 references that are in the map. First posted on arXiv 17 Sep 2024.

Builds on

29 of 35 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
17 more
  1. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  2. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  3. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  4. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  5. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  7. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  8. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  9. 2016Visual Dialogcited by 377
  10. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  11. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  12. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
  13. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  14. 2023UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankerscited by 37
  15. 2024RDRec: Rationale Distillation for LLM-based Recommendationcited by 21
  16. 2023Improving Audio Captioning Models with Fine-grained Audio Features, Text Embedding Supervision, and LLM Mix-up Augmentationcited by 19
  17. 2023Large-scale Language Model Rescoring on Long-form Datacited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Modelscited by 10
  2. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
  3. 2026SemantiCache: Efficient KV Cache Compression via Semantic Chunking and Clustered Mergingcited by 5
  4. 2026CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilationcited by 1
  5. 2025The silence of the weights: a structural pruning strategy for attention-based audio signal architectures with second order metricscited by 0
  6. 2025PatternKV: Flattening KV Representation Expands Quantization Headroomcited by 0