mlsanityData to 14 Oct 2026

Weak-to-Strong Knowledge Distillation Accelerates Visual Learning

arXiv 2604.15451 ↗First posted 16 Apr 2026cs.CVcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Large-scale visual learning is increasingly limited by training cost. Existing knowledge distillation methods transfer from a stronger teacher to a weaker student for compression or final-accuracy improvement. We instead investigate distillation to accelerate the training of strong students. We propose a generalizable plug-and-play recipe that freezes a weaker teacher, applies distillation only in early training, and turns it off once the student reaches and surpasses teacher-level performance. For ImageNet and CIFAR classification, this strategy reaches target thresholds much earlier, with up to 4.8 times speedup measured by epochs. We confirm that the method generalizes to other tasks and report 1.7 times epoch speedup for object detection on the COCO dataset, and 2.5 times earlier target-FID crossing for diffusion generation on the CIFAR-10 dataset, measured in steps. These findings validate our method as a universal speedup mechanism for visual learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 52 of its 61 references that are in the map. First posted on arXiv 16 Apr 2026.

Builds on

52 of 61 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Cumulative Spatial Knowledge Distillation for Vision Transformers17 Jul
  2. 2023Knowledge Distillation Performs Partial Variance Reduction27 May
  3. 2023Class Attention Transfer Based Knowledge Distillation25 Apr
  4. 2023Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning13 Apr
  5. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning8 Dec
  6. 2022DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers27 Apr
  7. 2022Decoupled Knowledge Distillation16 Mar
  8. 2022Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability10 Mar
  9. 2021A Fast Knowledge Distillation Framework for Visual Recognition2 Dec
  10. 2021Knowledge distillation: A good teacher is patient and consistent9 Jun

Other references · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2023Segment Anythingcited by 8,234
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  9. 2016Densely Connected Convolutional Networkscited by 6,720
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
30 more
  1. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  3. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  4. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  5. 2019Searching for MobileNetV3cited by 1,994
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  7. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  8. 2017Random Erasing Data Augmentationcited by 1,239
  9. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  10. 2019Relational Knowledge Distillationcited by 889
  11. 2019Contrastive Representation Distillationcited by 748
  12. 2022On Distillation of Guided Diffusion Modelscited by 725
  13. 2017Deep Mutual Learningcited by 628
  14. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  15. 2018Born Again Neural Networkscited by 546
  16. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  17. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 482
  18. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  19. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 410
  20. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  21. 2020Meta Pseudo Labelscited by 374
  22. 2021ResNet strikes back: An improved training procedure in timmcited by 368
  23. 2020Regularizing Class-wise Predictions via Self-knowledge Distillationcited by 144
  24. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 126
  25. 2023Posterior Distillation Samplingcited by 46
  26. 2024Vision Superalignment: Weak-to-Strong Generalization for Vision Foundation Modelscited by 28
  27. 2024Plug-and-Play Diffusion Distillationcited by 16
  28. 2024DeiT-LT Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasetscited by 14
  29. 2022Pixel-Wise Contrastive Distillationcited by 5
  30. 2024Weak-to-Strong 3D Object Detection with X-Ray Distillationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation
  2. 2025Scaling Laws for Data-Efficient Visual Transfer Learning
  3. 2022Prediction-Guided Distillation for Dense Object Detection
  4. 2025Competitive Distillation: A Simple Learning Strategy for Improving Visual Classification
  5. 2024On Good Practices for Task-Specific Distillation of Large Pretrained Visual Models
  6. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation
  7. 2023Understanding the Role of the Projector in Knowledge Distillation
  8. 2024AMD: Automatic Multi-step Distillation of Large-scale Vision Models