mlsanityData to 7 Oct 2026

Prune Your Model Before Distill It

arXiv 2109.14960 ↗First posted 30 Sep 2021cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation transfers the knowledge from a cumbersome teacher to a small student. Recent results suggest that the student-friendly teacher is more appropriate to distill since it provides more transferable knowledge. In this work, we propose the novel framework, "prune, then distill," that prunes the model first to make it more transferrable and then distill it to the student. We provide several exploratory examples where the pruned teacher teaches better than the original unpruned networks. We further show theoretically that the pruned teacher plays the role of regularizer in distillation, which reduces the generalization error. Based on this result, we propose a novel neural network compression scheme where the student network is formed based on the pruned teacher and then apply the "prune, then distill" strategy. The code is available at https://github.com/ososos888/prune-then-distill

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 45 of its 61 references that are in the map. First posted on arXiv 30 Sep 2021.

Builds on

45 of 61 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2016Wide Residual Networkscited by 3,057
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  12. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
33 more
  1. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  2. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  3. 2019Contrastive Representation Distillationcited by 734
  4. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  5. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  6. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  7. 2018Rethinking the Value of Network Pruningcited by 595
  8. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  10. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  11. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  12. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  13. 2019On the Efficacy of Knowledge Distillationcited by 377
  14. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 313
  15. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 291
  16. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  17. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  18. 2019MetaPruning: Meta Learning for Automatic Neural Network Channel Pruningcited by 226
  19. 2016Ternary Weight Networkscited by 195
  20. 2021Does Knowledge Distillation Really Work?cited by 187
  21. 2015Structured Pruning of Deep Convolutional Neural Networkscited by 148
  22. 2018Scalable Methods for 8-bit Training of Neural Networkscited by 139
  23. 2017Sobolev Training for Neural Networkscited by 136
  24. 2020Distilling Knowledge from Graph Convolutional Networkscited by 128
  25. 2020Layer-adaptive sparsity for the Magnitude-based Pruningcited by 122
  26. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  27. 2018Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layerscited by 112
  28. 2018Knowledge Transfer with Jacobian Matchingcited by 73
  29. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 58
  30. 2018Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameterscited by 56
  31. 2019DeepCABAC: A Universal Compression Algorithm for Deep Neural Networkscited by 26
  32. 2021BCNet: Searching for Network Width with Bilaterally Coupled Networkcited by 23
  33. 2021The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularizationcited by 6

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 6; 2024: 3.

12 papers.

  1. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  2. 2023Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Modelcited by 19
  3. 2022Join the High Accuracy Club on ImageNet with A Binary Neural Network Ticketcited by 13
  4. 2022Progressive Learning without Forgettingcited by 7
  5. 2023DSD2: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free?cited by 6
  6. 2024EPSD: Early Pruning with Self-Distillation for Efficient Model Compressioncited by 6
  7. 2024Can Students Beyond The Teacher? Distilling Knowledge from Teacher's Biascited by 5
  8. 2023RdimKD: Generic Distillation Paradigm by Dimensionality Reductioncited by 3
  9. 2024Attention-guided Feature Distillation for Semantic Segmentationcited by 3
  10. 2022InDistill: Information flow-preserving knowledge distillation for model compressioncited by 2
  11. 2023LAPP: Layer Adaptive Progressive Pruning for Compressing CNNs from Scratchcited by 2
  12. 2023The Quest of Finding the Antidote to Sparse Double Descentcited by 0