mlsanityData to 7 Oct 2026

Knowledge Distillation by On-the-Fly Native Ensemble

arXiv 1806.04606 ↗First posted 12 Jun 2018cs.CVcited by 186

Not on the map. Why?

Abstract (arXiv)

Knowledge distillation is effective to train small and generalisable network models for meeting the low-memory and fast running requirements. Existing offline distillation methods rely on a strong pre-trained teacher, which enables favourable knowledge discovery and transfer but requires a complex two-phase training procedure. Online counterparts address this limitation at the price of lacking a highcapacity teacher. In this work, we present an On-the-fly Native Ensemble (ONE) strategy for one-stage online distillation. Specifically, ONE trains only a single multi-branch network while simultaneously establishing a strong teacher on-the- fly to enhance the learning of target network. Extensive evaluations show that ONE improves the generalisation performance a variety of deep neural networks more significantly than alternative methods on four image classification dataset: CIFAR10, CIFAR100, SVHN, and ImageNet, whilst having the computational efficiency advantages.

Key facts. Cited by 186 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 47; 2022: 41; 2023: 38; 2024: 35; 2025: 20; 2026: 5.

186 papers.

Sort:

By area:

  1. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165
  2. 2021Patch Slimming for Efficient Vision Transformerscited by 131
  3. 2021Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labelscited by 117
  4. 2021Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognitioncited by 89
  5. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  6. 2023One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillationcited by 77
  7. 2022Nested Collaborative Learning for Long-Tailed Visual Recognitioncited by 63
  8. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 58
  9. 2021Online Knowledge Distillation for Efficient Pose Estimationcited by 55
  10. 2021A Survey on Green Deep Learningcited by 52
  11. 2021DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimationcited by 48
  12. 2021Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillationcited by 46
38 more
  1. 2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillationcited by 45
  2. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  3. 2021Mutual Contrastive Learning for Visual Representation Learningcited by 42
  4. 2022A Survey on Heterogeneous Federated Learningcited by 42
  5. 2021Confidence-Aware Multi-Teacher Knowledge Distillationcited by 39
  6. 2022Efficient Knowledge Distillation from Model Checkpointscited by 37
  7. 2022Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognitioncited by 36
  8. 2023A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentationcited by 33
  9. 2021MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworkscited by 32
  10. 2021Student Network Learning via Evolutionary Knowledge Distillationcited by 30
  11. 2022MixSKD: Self-Knowledge Distillation from Mixup for Image Recognitioncited by 30
  12. 2022Self-Distillation from the Last Mini-Batch for Consistency Regularizationcited by 29
  13. 2023DisWOT: Student Architecture Search for Distillation WithOut Trainingcited by 29
  14. 2021On the Reproducibility of Neural Network Predictionscited by 28
  15. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  16. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  17. 2021Online Multi-Granularity Distillation for GAN Compressioncited by 24
  18. 2022Monitored Distillation for Positive Congruent Depth Completioncited by 24
  19. 2021Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfercited by 23
  20. 2022Teacher-Student Architecture for Knowledge Learning: A Surveycited by 23
  21. 2021Distill on the Go: Online knowledge distillation in self-supervised learningcited by 22
  22. 2021Distilling a Powerful Student Model via Online Knowledge Distillationcited by 21
  23. 2021Handwritten Mathematical Expression Recognition via Attention Aggregation based Bi-directional Mutual Learningcited by 20
  24. 2022Contrastive Deep Supervisioncited by 20
  25. 2022Knowledge Condensation Distillationcited by 20
  26. 2022Integrating Object-aware and Interaction-aware Knowledge for Weakly Supervised Scene Graph Generationcited by 20
  27. 2023Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learningcited by 20
  28. 2023Improving Plasticity in Online Continual Learning via Collaborative Learningcited by 20
  29. 2021Evaluating Deep Graph Neural Networkscited by 19
  30. 2021LGD: Label-guided Self-distillation for Object Detectioncited by 19
  31. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 19
  32. 2022Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translationcited by 18
  33. 2023Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspectivecited by 17
  34. 2024MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolutioncited by 17
  35. 2024PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillationcited by 17
  36. 2021Self-distillation with Batch Knowledge Ensembling Improves ImageNet Classificationcited by 16
  37. 2023Pathologies of Predictive Diversity in Deep Ensemblescited by 16
  38. 2023KD-DLGAN: Data Limited Image Generation via Knowledge Distillationcited by 16

50 most cited of 186 All 186 (JSON)