Data to 7 Oct 2026

Structured Pruning Learns Compact and Accurate Models

arXiv 2204.00408 ↗First posted 1 Apr 2022cs.CL cs.LGcited by 167

Not on the map. Why?

Abstract (arXiv)

The growing size of neural language models has led to increased attention in model compression. The two predominant approaches are pruning, which gradually removes weights from a pre-trained model, and distillation, which trains a smaller compact model to match a larger one. Pruning methods can significantly reduce the model size but hardly achieve large speedups as distillation. However, distillation methods require large amounts of unlabeled data and are expensive to train. In this work, we propose a task-specific structured pruning method CoFi (Coarse- and Fine-grained Pruning), which delivers highly parallelizable subnetworks and matches the distillation methods in both accuracy and latency, without resorting to any unlabeled data. Our key insight is to jointly prune coarse-grained (e.g., layers) and fine-grained (e.g., heads and hidden units) modules, which controls the pruning decision of each parameter with masks of different granularity. We also devise a layerwise distillation strategy to transfer knowledge from unpruned to pruned models during optimization. Our experiments on GLUE and SQuAD datasets show that CoFi yields models with over 10x speedups with a small accuracy drop, showing its effectiveness and efficiency compared to previous pruning and distillation approaches.

Key facts. Cited by 167 papers in the map (data through 7 Oct 2026). Builds on 47 of its 61 references that are in the map. First posted on arXiv 1 Apr 2022.

Builds on

47 of 61 references.

References · 47

2017Attention Is All You Needcited by 59,932
2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
2015Distilling the Knowledge in a Neural Networkcited by 10,076
2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153

35 more

2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
2018Neural Network Acceptability Judgmentscited by 952
2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
2019Are Sixteen Heads Really Better than One?cited by 803
2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
2018Rethinking the Value of Network Pruningcited by 595
2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
2019Importance Estimation for Neural Network Pruningcited by 531
2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
2019MetaPruning: Meta Learning for Automatic Neural Network Channel Pruningcited by 226
2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 206
2019Structured Pruning of Large Language Modelscited by 201
2021Block Pruning For Faster Transformerscited by 176
2020Training with Quantization Noise for Extreme Model Compressioncited by 145
2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 95
2020EarlyBERT: Efficient BERT Training via Early-bird Lottery Ticketscited by 79
2020DMCP: Differentiable Markov Channel Pruning for Neural Networkscited by 68
2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
2019Improving Transformer Models by Reordering their Sublayerscited by 52
2021Differentiable Subset Pruning of Transformer Headscited by 46
2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40
2021AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Modelscited by 31
2021Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigmcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 59; 2024: 47; 2025: 28; 2026: 19.

167 papers.

Sort:

By area:

2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
2024A Survey on Efficient Inference for Large Language Modelscited by 197
2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
2023TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritancecited by 90
2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89

38 more

2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
2022Efficient Methods for Natural Language Processing: A Surveycited by 78
2024CLLMs: Consistency Large Language Modelscited by 71
2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
2024Finding Transformer Circuits with Edge Pruningcited by 58
2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 46
2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 45
2024A Survey on Transformer Compressioncited by 44
2024Everybody Prune Now: Structured Pruning of LLMs with only Forward Passescited by 42
2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 41
2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
2023Masked Structural Growth for 2x Faster Language Model Pre-trainingcited by 32
2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 32
2023What Matters In The Structured Pruning of Generative Language Models?cited by 30
2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 29
2024APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inferencecited by 29
2025Vision-Language Models for Edge Networks: A Comprehensive Surveycited by 29
2023Gradient-Free Structured Pruning with Unlabeled Datacited by 26
2023HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformerscited by 25
2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
2023Faster Causal Attention Over Large Sequences Through Sparse Flash Attentioncited by 24
2024DISP-LLM: Dimension-Independent Structural Pruning for Large Language Modelscited by 24
2023Selective Pre-training for Private Fine-tuningcited by 22
2022Efficiently Controlling Multiple Risks with Pareto Testingcited by 21
2022Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Modelscited by 20
2022A Survey for Efficient Open Domain Question Answeringcited by 20
2024Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generationcited by 20
2022MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attentioncited by 19
2023Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Modelscited by 19
2024OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decompositioncited by 19
2024Understanding Layer Significance in LLM Alignmentcited by 19

50 most cited of 167 All 167 (JSON)