mlsanityData to 13 Oct 2026

Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning

arXiv 2002.08307 ↗First posted 19 Feb 2020cs.CLcited by 184

Not on the map. Why?

Abstract (arXiv)

Pre-trained universal feature extractors, such as BERT for natural language processing and VGG for computer vision, have become effective methods for improving deep learning models without requiring more labeled data. While effective, feature extractors like BERT may be prohibitively large for some deployment scenarios. We explore weight pruning for BERT and ask: how does compression during pre-training affect transfer learning? We find that pruning affects transfer learning in three broad regimes. Low levels of pruning (30-40%) do not affect pre-training loss or transfer to downstream tasks at all. Medium levels of pruning increase the pre-training loss and prevent useful pre-training information from being transferred to downstream tasks. High levels of pruning additionally prevent models from fitting downstream datasets, leading to further degradation. Finally, we observe that fine-tuning BERT on a specific task does not improve its prunability. We conclude that BERT can be pruned once during pre-training rather than separately for each task without affecting performance.

Key facts. Cited by 184 papers in the map (data through 13 Oct 2026). First posted on arXiv 19 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 50; 2022: 48; 2023: 42; 2024: 24; 2025: 13; 2026: 7.

184 papers.

Sort:

By area:

  1. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,240
  2. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 522
  3. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  4. 2023A Survey on Model Compression for Large Language Modelscited by 268
  5. 2021I-BERT: Integer-only BERT Quantizationcited by 254
  6. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 210
  7. 2021Block Pruning For Faster Transformerscited by 176
  8. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  9. 2022Pretrained Language Models for Text Generation: A Surveycited by 122
  10. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  11. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 105
  12. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 93
38 more
  1. 2021Scalable Vision Transformers with Hierarchical Poolingcited by 84
  2. 2022Unified Visual Transformer Compressioncited by 79
  3. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  4. 2021Vision Transformer Pruningcited by 77
  5. 2022BiBERT: Accurate Fully Binarized BERTcited by 72
  6. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  7. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  8. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  9. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 66
  10. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  11. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 58
  12. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 56
  13. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  14. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  15. 2024Efficient Tuning and Inference for Large Language Models on Textual Graphscited by 54
  16. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 53
  17. 2021Differentiable Subset Pruning of Transformer Headscited by 46
  18. 2021Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinesecited by 45
  19. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  20. 2021Towards Efficient NLP: A Standard Evaluation and A Strong Baselinecited by 44
  21. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  22. 2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40
  23. 2023Fast as CHITA: Neural Network Pruning with Combinatorial Optimizationcited by 40
  24. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 39
  25. 2022CP-ViT: Cascade Vision Transformer Pruning via Progressive Sparsity Predictioncited by 39
  26. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  27. 2021NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Frameworkcited by 34
  28. 2022A Simple Hash-Based Early Exiting Approach For Language Understanding and Generationcited by 32
  29. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  30. 2021HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizationcited by 31
  31. 2023Can Public Large Language Models Help Private Cross-device Federated Learning?cited by 30
  32. 2022Efficient Fine-Tuning of BERT Models on the Edgecited by 29
  33. 2022Parameter-Efficient Sparsity for Large Language Models Fine-Tuningcited by 27
  34. 2021Scaling Laws for Deep Learningcited by 25
  35. 2023AccelTran: A Sparsity-Aware Accelerator for Dynamic Inference with Transformerscited by 24
  36. 2023Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Casescited by 23
  37. 2023Learned Thresholds Token Merging and Pruning for Vision Transformerscited by 21
  38. 2021Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigmcited by 20

50 most cited of 184 All 184 (JSON)