mlsanityData to 14 Oct 2026

Representation Consolidation for Training Expert Students

arXiv 2107.08039 ↗First posted 16 Jul 2021cs.CV cs.LGcited by 5

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Traditionally, distillation has been used to train a student model to emulate the input/output functionality of a teacher. A more useful goal than emulation, yet under-explored, is for the student to learn feature representations that transfer well to future tasks. However, we observe that standard distillation of task-specific teachers actually *reduces* the transferability of student representations to downstream tasks. We show that a multi-head, multi-task distillation method using an unlabeled proxy dataset and a generalist teacher is sufficient to consolidate representations from task-specific teacher(s) and improve downstream performance, outperforming the teacher(s) and the strong baseline of ImageNet pretrained features. Our method can also combine the representational knowledge of multiple teachers trained on one or multiple domains into a single model, whose representation is improved on all teachers' domain(s).

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 40 of its 63 references that are in the map. First posted on arXiv 16 Jul 2021.

Builds on

40 of 63 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2016Learning without Forgettingcited by 2,496
  7. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
  8. 2013Describing Textures in the Wildcited by 2,125
  9. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  10. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  11. 2019Contrastive Representation Distillationcited by 748
  12. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
28 more
  1. 2018Knockoff Nets: Stealing Functionality of Black-Box Modelscited by 292
  2. 2017Overcoming Catastrophic Forgetting by Incremental Moment Matchingcited by 243
  3. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  4. 2019Task2Vec: Task Embedding for Meta-Learningcited by 205
  5. 2019Data-Free Learning of Student Networkscited by 204
  6. 2017Data-Free Knowledge Distillation for Deep Neural Networkscited by 136
  7. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  8. 2019One-Shot Federated Learningcited by 125
  9. 2019Zero-Shot Knowledge Distillation in Deep Networkscited by 121
  10. 2019Data-Free Adversarial Distillationcited by 90
  11. 2020Generative Low-bitwidth Data Free Quantizationcited by 88
  12. 2019Knowledge Distillation from Internal Representationscited by 87
  13. 2019This dataset does not exist: training models from generated imagescited by 65
  14. 2020Object Detection with a Unified Label Space from Multiple Datasetscited by 65
  15. 2019The Knowledge Within: Methods for Data-Free Model Compressioncited by 54
  16. 2020Scalable Transfer Learning with Expert Modelscited by 49
  17. 2021A linearized framework and a new benchmark for model selection for fine-tuningcited by 41
  18. 2019The iMaterialist Fashion Attribute Datasetcited by 37
  19. 2019Knowledge Flow: Improve Upon Your Teacherscited by 36
  20. 2019FoodX-251: A Dataset for Fine-grained Food Classificationcited by 33
  21. 2020Data-Free Knowledge Amalgamation via Group-Stack Dual-GANcited by 32
  22. 2020Large-Scale Generative Data-Free Distillationcited by 29
  23. 2019Unifying Heterogeneous Classifiers with Distillationcited by 25
  24. 2019DeGAN : Data-Enriching GAN for Retrieving Representative Samples from a Trained Classifiercited by 21
  25. 2019Dream Distillation: A Data-Independent Model Compression Frameworkcited by 19
  26. 2018Unifying and Merging Well-trained Deep Neural Networks for Inference Stagecited by 8
  27. 2017Knowledge Concentration: Learning 100K Object Classifiers in a Single CNNcited by 6
  28. 2018A mixture model for aggregation of multiple pre-trained weak classifierscited by 1

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 2.

5 papers.

  1. 2021INTERN: A New Learning Paradigm Towards General Visioncited by 33
  2. 2022Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferabilitycited by 21
  3. 2022X-Learner: Learning Cross Sources and Tasks for Universal Visual Representationcited by 6
  4. 2023Distilling from Similar Tasks for Transfer Learning on a Budgetcited by 4
  5. 2023Self-Distillation for Gaussian Process Regression and Classificationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Multi-Level Feature Distillation of Joint Teachers Trained on Distinct Image Datasets
  2. 2021Fixing the Teacher-Student Knowledge Discrepancy in Distillation
  3. 2026Scaling Multi-Teacher Distillation for Digital Pathology
  4. 2024On Good Practices for Task-Specific Distillation of Large Pretrained Visual Models
  5. 2022Knowledge Distillation with the Reused Teacher Classifier
  6. 2021Teacher's pet: understanding and mitigating biases in distillation
  7. 2025SigLino: Efficient Multi-Teacher Distillation for Agglomerative Vision Foundation Models
  8. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation