mlsanityData to 7 Oct 2026

FitNets: Hints for Thin Deep Nets

arXiv 1412.6550 ↗First posted 19 Dec 2014cs.LG cs.NEcited by 1,619

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While depth tends to improve network performances, it also makes gradient-based training more difficult since deeper networks tend to be more non-linear. The recently proposed knowledge distillation approach is aimed at obtaining small and fast-to-execute models, and it has shown that a student network could imitate the soft output of a larger teacher network or ensemble of networks. In this paper, we extend this idea to allow the training of a student that is deeper and thinner than the teacher, using not only the outputs but also the intermediate representations learned by the teacher as hints to improve the training process and final performance of the student. Because the student intermediate hidden layer will generally be smaller than the teacher's intermediate hidden layer, additional parameters are introduced to map the student hidden layer to the prediction of the teacher hidden layer. This allows one to train deeper students that can generalize better or run faster, a trade-off that is controlled by the chosen student capacity. For example, on CIFAR-10, a deep student network with almost 10.4 times less parameters outperforms a larger, state-of-the-art teacher network.

Key facts. Cited by 1,619 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Dec 2014.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 291; 2022: 299; 2023: 314; 2024: 261; 2025: 215; 2026: 239.

1,619 papers.

Sort:

By area:

  1. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  3. 2021Going deeper with Image Transformerscited by 759
  4. 2023LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPScited by 420
  5. 2022Decoupled Knowledge Distillationcited by 380
  6. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  7. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speedcited by 344
  8. 2021Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networkscited by 313
  9. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  10. 2026Reinforcement Learning via Self-Distillationcited by 285
  11. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  12. 2021I-BERT: Integer-only BERT Quantizationcited by 250
38 more
  1. 2022Knowledge Distillation from A Stronger Teachercited by 204
  2. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  3. 2021Learning Distilled Collaboration Graph for Multi-Agent Perceptioncited by 194
  4. 2021Does Knowledge Distillation Really Work?cited by 187
  5. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 187
  6. 2023Zero-Shot Composed Image Retrieval with Textual Inversioncited by 171
  7. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165
  8. 2021Towards Understanding Knowledge Distillationcited by 158
  9. 2024Distilling System 2 into System 1cited by 152
  10. 2025Streaming 4D Visual Geometry Transformercited by 147
  11. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 140
  12. 2021Focal and Global Knowledge Distillation for Detectorscited by 138
  13. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  14. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  15. 2022Learning by Distilling Contextcited by 128
  16. 2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
  17. 2023Dataset Distillation: A Comprehensive Reviewcited by 126
  18. 2022Diffusion Probabilistic Model Made Slimcited by 121
  19. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 121
  20. 2022Self-Sustaining Representation Expansion for Non-Exemplar Class-Incremental Learningcited by 118
  21. 2022Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentationcited by 118
  22. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 117
  23. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  24. 2021General Instance Distillation for Object Detectioncited by 113
  25. 2022Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidationcited by 113
  26. 2022Curriculum Temperature for Knowledge Distillationcited by 110
  27. 2022Masked Generative Distillationcited by 107
  28. 2022Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformercited by 106
  29. 2021Distilling Object Detectors via Decoupled Featurescited by 105
  30. 2021Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillationcited by 104
  31. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  32. 2024Logit Standardization in Knowledge Distillationcited by 101
  33. 2021Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signalscited by 100
  34. 2023Diffusion Model as Representation Learnercited by 95
  35. 2021Trustworthy AI: A Computational Perspectivecited by 93
  36. 2022Membership Inference Attacks by Exploiting Loss Trajectorycited by 93
  37. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 91
  38. 2023Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Trainingcited by 91

50 most cited of 1,619 All 1,619 (JSON)