mlsanityData to 7 Oct 2026

VkD: Improving Knowledge Distillation using Orthogonal Projections

arXiv 2403.06213 ↗First posted 10 Mar 2024cs.CV cs.AIcited by 24

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation is an effective method for training small and efficient deep learning models. However, the efficacy of a single method can degenerate when transferring to other tasks, modalities, or even other architectures. To address this limitation, we propose a novel constrained feature distillation method. This method is derived from a small set of core principles, which results in two emerging components: an orthogonal projection and a task-specific normalisation. Equipped with both of these components, our transformer models can outperform all previous methods on ImageNet and reach up to a 4.4% relative improvement over the previous state-of-the-art methods. To further demonstrate the generality of our method, we apply it to object detection and image generation, whereby we obtain consistent and substantial performance improvements over state-of-the-art. Code and models are publicly available: https://github.com/roymiles/vkd

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 70 of its 91 references that are in the map. First posted on arXiv 10 Mar 2024.

Builds on

70 of 91 references.

References · 70

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  4. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  5. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  6. 2020Designing Network Design Spacescited by 954
  7. 2019Relational Knowledge Distillationcited by 873
  8. 2021Going deeper with Image Transformerscited by 759
  9. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  10. 2019Similarity-Preserving Knowledge Distillationcited by 507
  11. 2022DeiT III: Revenge of the ViTcited by 417
  12. 2022Decoupled Knowledge Distillationcited by 380
  13. 2019On the Efficacy of Knowledge Distillationcited by 377
  14. 2020Differentiable Augmentation for Data-Efficient GAN Trainingcited by 361
  15. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  16. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  17. 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 291
  18. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  19. 2019Correlation Congruence for Knowledge Distillationcited by 280
  20. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  21. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  22. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  23. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  24. 2015Unifying distillation and privileged informationcited by 218
  25. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 209
  26. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  27. 2020Knowledge Distillation Meets Self-Supervisioncited by 142
  28. 2020BYOL works even without batch statisticscited by 95
  29. 2019XNOR-Net++: Improved Binary Neural Networkscited by 94
  30. 2019Deep Learning for Audio Signal Processingcited by 92
  31. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 82
  32. 2018Knowledge Transfer with Jacobian Matchingcited by 73
  33. 2019Riemannian batch normalization for SPD neural networkscited by 72
  34. 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
  35. 2020DMCP: Differentiable Markov Channel Pruning for Neural Networkscited by 68
  36. 2021Contrastive Model Inversion for Data-Free Knowledge Distillationcited by 66
  37. 2020OBoW: Online Bag-of-Visual-Words Generation for Self-Supervised Learningcited by 65
  38. 2022DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformerscited by 65

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 8; 2026: 9.

24 papers.

Sort:

By area:

  1. 2026ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Modelscited by 11
  2. 2024A Comprehensive Review of Knowledge Distillation in Computer Visioncited by 10
  3. 2025Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learningcited by 5
  4. 2024Dual-Head Knowledge Distillation: Enhancing Logits Utilization with an Auxiliary Headcited by 4
  5. 2025HDC: Hierarchical Distillation for Multi-level Noisy Consistency in Semi-Supervised Fetal Ultrasound Segmentationcited by 3
  6. 2025ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Imagescited by 2
  7. 2026Distilling Lightweight Domain Experts from Large ML Models by Identifying Relevant Subspacescited by 2
  8. 2024Teaching with Uncertainty: Unleashing the Potential of Knowledge Distillation in Object Detectioncited by 1
  9. 2024Relational Representation Distillationcited by 1
  10. 2025Transferable Model-agnostic Vision-Language Model Adaptation for Efficient Weak-to-Strong Generalizationcited by 1
  11. 2025From Per-Image Low-Rank to Encoding Mismatch: Rethinking Feature Distillation in Vision Transformerscited by 1
  12. 2026Investigating the Robustness of Subtask Distillation under Spurious Correlationcited by 1
12 more
  1. 2026Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Bettercited by 1
  2. 2024Learning to Project for Cross-Task Knowledge Distillationcited by 0
  3. 2024Cross-Domain Knowledge Distillation for Low-Resolution Human Pose Estimationcited by 0
  4. 2024Preserving Angles Improves Feature Distillationcited by 0
  5. 2025MoKD: Multi-Task Optimization for Knowledge Distillationcited by 0
  6. 2025Knowledge distillation through geometry-aware representational alignmentcited by 0
  7. 2025MobiAct: Efficient MAV Action Recognition Using MobileNetV4 with Contrastive Learning and Knowledge Distillationcited by 0
  8. 2026Integrating Knowledge Distillation Methods: A Sequential Multi-Stage Frameworkcited by 0
  9. 2026BrainDistill: Implantable Motor Decoding with Task-Specific Knowledge Distillationcited by 0
  10. 2026REDistill: Robust Estimator Distillation for Balancing Robustness and Efficiencycited by 0
  11. 2026Heterogeneous Knowledge Distillation via Geometry Decoupling and Momentum-Aware Gradient Regulationcited by 0
  12. 2026The Devil is in the Spectrum Bias: Spectrum-Balanced Feature Matching for Robust Representation Distillationcited by 0