# Gradient-Free Structured Pruning with Unlabeled Data

arXiv: https://arxiv.org/abs/2303.04185 · HTML: https://mlsanity.com/paper/2303.04185/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 72 of its 100 references that are in the map.
- First posted on arXiv 7 Mar 2023.

## Builds on (most cited first, 30 of 72)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned](https://mlsanity.com/paper/1905.09418/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT](https://mlsanity.com/paper/1909.05840/) (2019)
- [DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference](https://mlsanity.com/paper/2004.12993/) (2020)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [The Lottery Ticket Hypothesis for Pre-trained BERT Networks](https://mlsanity.com/paper/2007.12223/) (2020)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [DynaBERT: Dynamic BERT with Adaptive Width and Depth](https://mlsanity.com/paper/2004.04037/) (2020)

## Built on by (most cited first, 26 of 26)

- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods](https://mlsanity.com/paper/2402.02834/) (2024)
- [Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes](https://mlsanity.com/paper/2402.05406/) (2024)
- [A Survey of Lottery Ticket Hypothesis](https://mlsanity.com/paper/2403.04861/) (2024)
- [Pruning Large Language Models with Semi-Structural Adaptive Sparse Training](https://mlsanity.com/paper/2407.20584/) (2024)
- [A Comprehensive Survey of Compression Algorithms for Language Models](https://mlsanity.com/paper/2401.15347/) (2024)
- [Accurate Retraining-free Pruning for Pretrained Encoder-based Language Models](https://mlsanity.com/paper/2308.03449/) (2023)
- [Sliding-Window Merging for Compacting Patch-Redundant Layers in LLMs](https://mlsanity.com/paper/2502.19159/) (2025)
- [The Right to be Forgotten in Pruning: Unveil Machine Unlearning on Sparse Models](https://mlsanity.com/paper/2507.18725/) (2025)
- [Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layers](https://mlsanity.com/paper/2507.23362/) (2025)
- [rm SP3: Enhancing Structured Pruning via PCA Projection](https://mlsanity.com/paper/2308.16475/) (2023)
- [MINI-LLM: Memory-Efficient Structured Pruning for Large Language Models](https://mlsanity.com/paper/2407.11681/) (2024)
- [Reconstruct the Pruned Model without Any Retraining](https://mlsanity.com/paper/2407.13331/) (2024)
- [Prune-then-Quantize or Quantize-then-Prune? Understanding the Impact of Compression Order in Joint Model Compression](https://mlsanity.com/paper/2603.18426/) (2026)
- [CORP: Closed-Form One-shot Representation-Preserving Structured Pruning for Transformers](https://mlsanity.com/paper/2602.05243/) (2026)
- [Large Language Model Pruning](https://mlsanity.com/paper/2406.00030/) (2024)
- [Harnessing Input-Adaptive Inference for Efficient VLN](https://mlsanity.com/paper/2508.09262/) (2025)
- [SlimDiff: Training-Free, Activation-Guided Hands-free Slimming of Diffusion Models](https://mlsanity.com/paper/2509.21498/) (2025)
- [CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Models](https://mlsanity.com/paper/2509.25996/) (2025)
- [Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation](https://mlsanity.com/paper/2510.00192/) (2025)
- [Frustratingly Easy Task-aware Pruning for Large Language Models](https://mlsanity.com/paper/2510.22489/) (2025)
- [Not All Attention is Needed: Parameter and Computation Efficient Transfer Learning for Multi-modal Large Language Models](https://mlsanity.com/paper/2403.15226/) (2024)
- [Inducing Semi-Structured Sparsity by Masking for Efficient Model Inference in Convolutional Networks](https://mlsanity.com/paper/2411.00288/) (2024)
- [Efficient Large Language Models with Zero-Shot Adjustable Acceleration](https://mlsanity.com/paper/2509.01190/) (2025)
- [Compositionality Emerges in a Narrow Depth-Connectivity Regime: Architecture Constraints and Solution Manifolds](https://mlsanity.com/paper/2606.19941/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have achieved great success in solving difficult tasks across many domains, but such success comes with a high computation cost, and inference latency. As developers and third parties customize these models, the need to provide efficient inference has increased. Many efforts have attempted to reduce inference cost through model compression techniques such as pruning and distillation. However, these techniques either require labeled data, or are time-consuming as they require the compressed model to be retrained to regain accuracy. In this paper, we propose a gradient-free structured pruning framework that uses only unlabeled data. An evaluation on the GLUE and SQuAD benchmarks using BERTBASE and DistilBERT illustrates the effectiveness of the proposed approach. By only using the weights of the pre-trained model and unlabeled data, in a matter of a few minutes on a single GPU, up to 40% of the original FLOP count can be reduced with less than a 4% accuracy loss across all tasks considered.