mlsanityData to 13 Oct 2026

BlockPruner: Fine-grained Pruning for Large Language Models

arXiv 2406.10594 ↗First posted 15 Jun 2024cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

With the rapid growth in the size and complexity of large language models (LLMs), the costs associated with their training and inference have escalated significantly. Research indicates that certain layers in LLMs harbor substantial redundancy, and pruning these layers has minimal impact on the overall performance. While various layer pruning methods have been developed based on this insight, they generally overlook the finer-grained redundancies within the layers themselves. In this paper, we delve deeper into the architecture of LLMs and demonstrate that finer-grained pruning can be achieved by targeting redundancies in multi-head attention (MHA) and multi-layer perceptron (MLP) blocks. We propose a novel, training-free structured pruning approach called BlockPruner. Unlike existing layer pruning methods, BlockPruner segments each Transformer layer into MHA and MLP blocks. It then assesses the importance of these blocks using perplexity measures and applies a heuristic search for iterative pruning. We applied BlockPruner to LLMs of various sizes and architectures and validated its performance across a wide range of downstream tasks. Experimental results show that BlockPruner achieves more granular and effective pruning compared to state-of-the-art baselines.

Key facts. Cited by 36 papers in the map (data through 13 Oct 2026). Builds on 26 of its 43 references that are in the map. First posted on arXiv 15 Jun 2024.

Builds on

26 of 43 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,388
  8. 2016Pointer Sentinel Mixture Modelscited by 3,047
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,133
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,103
14 more
  1. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 854
  2. 2023Baichuan 2: Open Large-scale Language Modelscited by 668
  3. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 646
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 522
  5. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 409
  6. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 381
  7. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 375
  8. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  9. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  10. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 125
  11. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 104
  12. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  13. 2024BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocationcited by 45
  14. 2023Weight subcloning: direct initialization of transformers using larger pretrained onescited by 30

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 4; 2025: 12; 2026: 20.

36 papers.

Sort:

By area:

  1. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
  2. 2024Reassessing Layer Pruning in LLMs: New Insights and Methodscited by 24
  3. 20252SSP: A Two-Stage Framework for Structured Pruning of LLMscited by 12
  4. 2026SafeNeuron: Neuron-Level Safety Alignment for Large Language Modelscited by 9
  5. 2025Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Modelscited by 7
  6. 2025DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMscited by 6
  7. 2026AA-SVD : Anchored and Adaptive SVD for Large Language Model Compressioncited by 6
  8. 2025COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximationcited by 5
  9. 2025When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMscited by 5
  10. 2026Rethinking Layer Relevance in Large Language Models Beyond Cosine Similaritycited by 4
  11. 2025RAP: Runtime Adaptive Pruning for LLM Inferencecited by 3
  12. 2025Accurate Sublayer Pruning for Large Language Models by Exploiting Latency and Tunability Informationcited by 2
24 more
  1. 2026It Takes a MAESTRO To Prune Bad Expertscited by 2
  2. 2024Research on Personalized Compression Algorithm for Pre-trained Models Based on Homomorphic Entropy Increasecited by 1
  3. 2025Federated Multimodal Learning with Dual Adapters and Selective Pruning for Communication and Computational Efficiencycited by 1
  4. 2025Mosaic: Composite Projection Pruning for Resource-efficient LLMscited by 1
  5. 2025High-Layer Attention Pruning with Rescalingcited by 1
  6. 2026Detect Anything in Real Time: From Single-Prompt Segmentation to Multi-Class Detectioncited by 1
  7. 2026No Free Swap: Protocol-Dependent Layer Redundancy in Transformerscited by 1
  8. 2026Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomycited by 1
  9. 2024FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routingcited by 0
  10. 2025AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMscited by 0
  11. 2025Beyond Uniform SVD:Dual-Level Optimization across Columns and Modules for LLM Compressioncited by 0
  12. 2026How Much of a Model Do We Need? Redundancy and Slimmability in Remote Sensing Foundation Modelscited by 0
  13. 2026GradMAP: Faster Layer Pruning with Gradient Metric and Projection Compensationcited by 0
  14. 2026Towards a Data-Parameter Correspondence for LLMs: A Preliminary Discussioncited by 0
  15. 2026A Game Theoretic Free Energy Analysis of Higher Order Synergy in Attention Heads of Large Language Modelscited by 0
  16. 2026TAPIOCA: Why Task- Aware Pruning Improves OOD model Capabilitycited by 0
  17. 2026Prune, Update and Trim: Robust Structured Pruning for Large Language Modelscited by 0
  18. 2026S4oP: Operator-level Pruning of Structured State Space Models for Resource-Constrained Devicescited by 0
  19. 2026Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimizationcited by 0
  20. 2026TriSP: Tri-Signal Structured Pruning for Large Language Modelscited by 0
  21. 2026Mask2Shield: Strengthening LLM Safety against Neuron-Pruning Attackscited by 0
  22. 2026WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruningcited by 0
  23. 2026LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compressioncited by 0
  24. 2026ReCal: Calibrating Structured Pruning for On-Policy Distillation Recoverycited by 0