mlsanityData to 7 Oct 2026

Multi-Modality Distillation via Learning the teacher's modality-level Gram Matrix

arXiv 2112.11447 ↗First posted 21 Dec 2021cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

In the context of multi-modality knowledge distillation research, the existing methods was mainly focus on the problem of only learning teacher final output. Thus, there are still deep differences between the teacher network and the student network. It is necessary to force the student network to learn the modality relationship information of the teacher network. To effectively exploit transfering knowledge from teachers to students, a novel modality relation distillation paradigm by modeling the relationship information among different modality are adopted, that is learning the teacher modality-level Gram Matrix.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 21 Dec 2021.

Builds on

28 of 41 references.

References · 28

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  4. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  5. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  6. 2018Rethinking the Value of Network Pruningcited by 595
  7. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  8. 2018Born Again Neural Networkscited by 539
  9. 2013Do Deep Nets Really Need to be Deep?cited by 466
  10. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
  11. 2018Model compression via distillation and quantizationcited by 287
  12. 2021Towards Understanding Knowledge Distillationcited by 158
16 more
  1. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  2. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  3. 2021Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillationcited by 104
  4. 2018Learning to Navigate for Fine-grained Classificationcited by 103
  5. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 100
  6. 2019Knowledge Distillation from Internal Representationscited by 87
  7. 2020Understanding and Improving Knowledge Distillationcited by 83
  8. 2017Adaptive Quantization for Deep Neural Networkcited by 67
  9. 2018Label Refinery: Improving ImageNet Classification through Label Progressioncited by 65
  10. 2018Attention-based Ensemble for Deep Metric Learningcited by 56
  11. 2020Semantics-aware Adaptive Knowledge Distillation for Sensor-to-Vision Action Recognitioncited by 37
  12. 2016Deep Model Compression: Distilling Knowledge from Noisy Teacherscited by 35
  13. 2020Efficient Knowledge Distillation for RNN-Transducer Modelscited by 29
  14. 2021Lipschitz Continuity Guided Knowledge Distillationcited by 19
  15. 2021Revisiting Knowledge Distillation: An Inheritance and Exploration Frameworkcited by 15
  16. 2020Simplified TinyBERT: Knowledge Distillation for Document Retrievalcited by 10

Built on by

Not cited yet.