mlsanityData to 7 Oct 2026

LumiNet: Perception-Driven Knowledge Distillation via Statistical Logit Calibration

arXiv 2310.03669 ↗First posted 5 Oct 2023cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

In the knowledge distillation literature, feature-based methods have dominated due to their ability to effectively tap into extensive teacher models. In contrast, logit-based approaches, which aim to distill "dark knowledge" from teachers, typically exhibit inferior performance compared to feature-based methods. To bridge this gap, we present LumiNet, a novel knowledge distillation algorithm designed to enhance logit-based distillation. We introduce the concept of "perception", aiming to calibrate logits based on the model's representation capability. This concept addresses overconfidence issues in the logit-based distillation method while also introducing a novel method to distill knowledge from the teacher. It reconstructs the logits of a sample/instances by considering relationships with other samples in the batch. LumiNet excels on benchmarks like CIFAR-100, ImageNet, and MSCOCO, outperforming the leading feature-based methods, e.g., compared to KD with ResNet18 and MobileNetV2 on ImageNet, it shows improvements of 1.5% and 2.05%, respectively. Codes are available at https://github.com/ismail31416/LumiNet.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 55 of its 78 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

55 of 78 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  9. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  12. 2014FitNets: Hints for Thin Deep Netscited by 1,619
38 more
  1. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  2. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  3. 2019Relational Knowledge Distillationcited by 873
  4. 2016Sequence-Level Knowledge Distillationcited by 761
  5. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  6. 2019Contrastive Representation Distillationcited by 734
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  8. 2017Deep Mutual Learningcited by 621
  9. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  10. 2018Born Again Neural Networkscited by 539
  11. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  12. 2019Similarity-Preserving Knowledge Distillationcited by 507
  13. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  14. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
  15. 2022Decoupled Knowledge Distillationcited by 380
  16. 2019On the Efficacy of Knowledge Distillationcited by 377
  17. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  18. 2018How Does Batch Normalization Help Optimization?cited by 355
  19. 2020Calibrating Deep Neural Networks using Focal Losscited by 353
  20. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  21. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 312
  22. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  23. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  24. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  25. 2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 251
  26. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  27. 2019Distilling Object Detectors with Fine-grained Feature Imitationcited by 188
  28. 2020Cross-Layer Distillation with Semantic Calibrationcited by 141
  29. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  30. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 118
  31. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  32. 2021Fair Feature Distillation for Visual Recognitioncited by 59
  33. 2023Knowledge Diffusion for Distillationcited by 51
  34. 2021Efficient Medical Image Segmentation Based on Knowledge Distillationcited by 43
  35. 2021Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-guided Feature Imitationcited by 40
  36. 2021Distilling Linguistic Context for Language Model Compressioncited by 33
  37. 2022Locality Guidance for Improving Vision Transformers on Tiny Datasetscited by 33
  38. 2023Dual Focal Loss for Calibrationcited by 33

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  2. 2026SGD-Based Knowledge Distillation with Bayesian Teachers: Theory and Guidelinescited by 0
  3. 2026Cognitive-Uncertainty Guided Knowledge Distillation for Accurate Classification of Student Misconceptionscited by 0