mlsanityData to 7 Oct 2026

Structural Pruning of Pre-trained Language Models via Neural Architecture Search

arXiv 2405.02267 ↗First posted 3 May 2024cs.LG cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models (PLM), for example BERT or RoBERTa, mark the state-of-the-art for natural language understanding task when fine-tuned on labeled data. However, their large size poses challenges in deploying them for inference in real-world applications, due to significant GPU memory requirements and high inference latency. This paper explores neural architecture search (NAS) for structural pruning to find sub-parts of the fine-tuned network that optimally trade-off efficiency, for example in terms of model size or latency, and generalization performance. We also show how we can utilize more recently developed two-stage weight-sharing NAS approaches in this setting to accelerate the search process. Unlike traditional pruning methods with fixed thresholds, we propose to adopt a multi-objective approach that identifies the Pareto optimal set of sub-networks, allowing for a more flexible and automated compression process.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 3 May 2024.

Builds on

35 of 44 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2016Layer Normalizationcited by 5,579
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  11. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
23 more
  1. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  2. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  3. 2019Are Sixteen Heads Really Better than One?cited by 803
  4. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  5. 2020What is the State of Neural Network Pruning?cited by 578
  6. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  7. 2017Large-Scale Evolution of Image Classifierscited by 348
  8. 2018Slimmable Neural Networkscited by 301
  9. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  10. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  11. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  12. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 219
  13. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 187
  14. 2020Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimizationcited by 176
  15. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  16. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  17. 2021How Powerful are Performance Predictors in Neural Architecture Search?cited by 88
  18. 2020AttentiveNAS: Improving Neural Architecture Search via Attentive Samplingcited by 65
  19. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 55
  20. 2020AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Searchcited by 51
  21. 2021Multi-objective Asynchronous Successive Halvingcited by 22
  22. 2020Exploring the Loss Landscape in Neural Architecture Searchcited by 20
  23. 2020Model-based Asynchronous Hyperparameter and Neural Architecture Searchcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2024HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Modelscited by 10
  2. 2025Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systemscited by 6
  3. 2025GPTailor: Large Language Model Pruning Through Layer Cutting and Stitchingcited by 3
  4. 2025Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillationcited by 1
  5. 2026Greedy-Gnorm: A Gradient Matrix Norm-Based Alternative to Attention Entropy for Head Pruningcited by 1