mlsanityData to 7 Oct 2026

Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigm

arXiv 2110.08190 ↗First posted 15 Oct 2021cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

Conventional wisdom in pruning Transformer-based language models is that pruning reduces the model expressiveness and thus is more likely to underfit rather than overfit. However, under the trending pretrain-and-finetune paradigm, we postulate a counter-traditional hypothesis, that is: pruning increases the risk of overfitting when performed at the fine-tuning phase. In this paper, we aim to address the overfitting problem and improve pruning performance via progressive knowledge distillation with error-bound properties. We show for the first time that reducing the risk of overfitting can help the effectiveness of pruning under the pretrain-and-finetune paradigm. Ablation studies and experiments on the GLUE benchmark show that our method outperforms the leading competitors across different tasks.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 36 of its 47 references that are in the map. First posted on arXiv 15 Oct 2021.

Builds on

36 of 47 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  9. 2017Decoupled Weight Decay Regularizationcited by 1,722
  10. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
24 more
  1. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  2. 2018Neural Network Acceptability Judgmentscited by 952
  3. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  4. 2020What is the State of Neural Network Pruning?cited by 578
  5. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  6. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  7. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  8. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  9. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  10. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  11. 2019Fast Sparse ConvNetscited by 86
  12. 2020Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficientcited by 81
  13. 2021Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Frameworkcited by 52
  14. 2019AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Ratescited by 49
  15. 2021Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigmcited by 40
  16. 2020Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruningcited by 39
  17. 2020Against Membership Inference Attack: Pruning is All You Needcited by 32
  18. 2019Reweighted Proximal Pruning for Large-Scale Language Representationcited by 29
  19. 2019Few Shot Network Compression via Cross Distillationcited by 26
  20. 2020Progressive Network Grafting for Few-Shot Knowledge Distillationcited by 25
  21. 2020NPAS: A Compiler-aware Framework of Unified Network Pruning and Architecture Search for Beyond Real-Time Mobile Accelerationcited by 16
  22. 2020Meta-Learning with Network Pruningcited by 14
  23. 2019Sparsity through evolutionary pruning prevents neuronal networks from overfittingcited by 11
  24. 2021Towards Extremely Compact RNNs for Video Recognition with Fully Decomposed Hierarchical Tucker Structurecited by 8

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 9; 2023: 7; 2024: 1; 2025: 2.

20 papers.

Sort:

By area:

  1. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  2. 2023Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matricescited by 72
  3. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  4. 2023AutoReP: Automatic ReLU Replacement for Fast Private Network Inferencecited by 29
  5. 2023Accel-GCN: High-Performance GPU Accelerator Design for Graph Convolution Networkscited by 22
  6. 2022PROD: Progressive Distillation for Dense Retrievalcited by 20
  7. 2022Towards Sparsification of Graph Neural Networkscited by 18
  8. 2022Dynamic Sparse Training via Balancing the Exploration-Exploitation Trade-offcited by 15
  9. 2022An Automatic and Efficient BERT Pruning for Edge AI Systemscited by 13
  10. 2021Robustness Challenges in Model Distillation and Pruning for Natural Language Understandingcited by 12
  11. 2023Accurate Neural Network Pruning Requires Rethinking Sparse Optimizationcited by 10
  12. 2025Scaling Down Text Encoders of Text-to-Image Diffusion Modelscited by 8
8 more
  1. 2022S4: a High-sparsity, High-performance AI Acceleratorcited by 3
  2. 2023PruMUX: Augmenting Data Multiplexing with Model Compressioncited by 3
  3. 2025CoVE: Compressed Vocabulary Expansion Makes Better LLM-based Recommender Systemscited by 3
  4. 2022SD-Conv: Towards the Parameter-Efficiency of Dynamic Convolutioncited by 2
  5. 2024ElastiFormer: Learned Redundancy Reduction in Transformer via Self-Distillationcited by 1
  6. 2022Efficient Traffic State Forecasting using Spatio-Temporal Network Dependencies: A Sparse Graph Neural Network Approachcited by 0
  7. 2023Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Methodcited by 0
  8. 2023Semi-supervised News Discourse Profiling with Contrastive Learningcited by 0