mlsanityData to 7 Oct 2026

Knowledge Distillation By Sparse Representation Matching

arXiv 2103.17012 ↗First posted 31 Mar 2021cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Knowledge Distillation refers to a class of methods that transfers the knowledge from a teacher network to a student network. In this paper, we propose Sparse Representation Matching (SRM), a method to transfer intermediate knowledge obtained from one Convolutional Neural Network (CNN) to another by utilizing sparse representation learning. SRM first extracts sparse representations of the hidden features of the teacher CNN, which are then used to generate both pixel-level and image-level labels for training intermediate feature maps of the student network. We formulate SRM as a neural processing block, which can be efficiently optimized using stochastic gradient descent and integrated into any CNN in a plug-and-play manner. Our experiments demonstrate that SRM is robust to architectural differences between the teacher and student networks, and outperforms other KD techniques across several datasets.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 40 of its 64 references that are in the map. First posted on arXiv 31 Mar 2021.

Builds on

40 of 64 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  8. 2018YOLOv3: An Incremental Improvementcited by 2,652
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  10. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  11. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  12. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
28 more
  1. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  2. 2019Relational Knowledge Distillationcited by 873
  3. 2019Contrastive Representation Distillationcited by 734
  4. 2013Do Deep Nets Really Need to be Deep?cited by 466
  5. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  6. 2019Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processingcited by 411
  7. 2019On the Efficacy of Knowledge Distillationcited by 377
  8. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  9. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  10. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  11. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  12. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  13. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  14. 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
  15. 2016A survey of sparse representation: algorithms and applicationscited by 58
  16. 2017Temporal Attention augmented Bilinear Network for Financial Time-Series Data Analysiscited by 46
  17. 2017Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networkscited by 21
  18. 2020Data Normalization for Bilinear Structures in High-Frequency Financial Time-seriescited by 16
  19. 2018Few-shot learning of neural networks from scratch by pseudo example optimizationcited by 15
  20. 2018Heterogeneous Multilayer Generalized Operational Perceptroncited by 14
  21. 2019Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillationcited by 10
  22. 2017Improving Efficiency in Convolutional Neural Network with Multilinear Filterscited by 8
  23. 2019QUEST: Quantized embedding space for transferring knowledgecited by 7
  24. 2020Attention-based Neural Bag-of-Features Learning for Sequence Datacited by 7
  25. 2017Automated Pruning for Deep Neural Network Compressioncited by 6
  26. 2019Multilinear Compressive Learningcited by 5
  27. 2020Multilinear Compressive Learning with Prior Knowledgecited by 3
  28. 2020Performance Indicator in Multilinear Compressive Learningcited by 3

Built on by

Not cited yet.