mlsanityData to 14 Oct 2026

Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning

arXiv 2012.09816 ↗First posted 17 Dec 2020cs.LG cs.NE math.OCcited by 297

Not on the map. Why?

Cited by 65 in Knowledge distillation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We formally study how ensemble of deep learning models can improve test accuracy, and how the superior performance of ensemble can be distilled into a single model using knowledge distillation. We consider the challenging case where the ensemble is simply an average of the outputs of a few independently trained neural networks with the SAME architecture, trained using the SAME algorithm on the SAME data set, and they only differ by the random seeds used in the initialization. We show that ensemble/knowledge distillation in Deep Learning works very differently from traditional learning theory (such as boosting or NTKs, neural tangent kernels). To properly understand them, we develop a theory showing that when data has a structure we refer to as ``multi-view'', then ensemble of independently trained neural networks can provably improve test accuracy, and such superior test accuracy can also be provably distilled into a single model by training a single model to match the output of the ensemble instead of the true label. Our result sheds light on how ensemble works in deep learning in a way that is completely different from traditional theorems, and how the ``dark knowledge'' is hidden in the outputs of the ensemble and can be used in distillation. In the end, we prove that self-distillation can also be viewed as implicitly combining ensemble and knowledge distillation to improve test accuracy.

Key facts. Cited by 297 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 32; 2022: 60; 2023: 71; 2024: 57; 2025: 53; 2026: 24.

297 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  2. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  3. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  4. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  6. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  7. 2023Mitigating the Alignment Tax of RLHFcited by 183
  8. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  9. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 156
  10. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 125
  11. 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 122
  12. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
38 more
  1. 2021Assessing Generalization of SGD via Disagreementcited by 117
  2. 2023In-Context Convergence of Transformerscited by 115
  3. 2024TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensemblingcited by 113
  4. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
  5. 2022Vision Transformers provably learn spatial structurecited by 93
  6. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
  7. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  8. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  9. 2023Boosting Multi-modal Model Performance with Adaptive Gradient Modulationcited by 73
  10. 2024Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaborationcited by 67
  11. 2023On Uni-Modal Feature Learning in Supervised Multi-Modal Learningcited by 53
  12. 2021S-LIME: Stabilized-LIME for Model Explanationcited by 52
  13. 2022Towards Understanding Mixture of Experts in Deep Learningcited by 52
  14. 2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 49
  15. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  16. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 45
  17. 2021Improving Multi-Modal Learning with Uni-Modal Teacherscited by 41
  18. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  19. 2023EmotionGesture: Audio-Driven Diverse Emotional Co-Speech 3D Gesture Generationcited by 40
  20. 2022Fortuitous Forgetting in Connectionist Networkscited by 39
  21. 2023Robust Learning with Progressive Data Expansion Against Spurious Correlationcited by 39
  22. 2022The Mechanism of Prediction Head in Non-contrastive Self-supervised Learningcited by 38
  23. 2023Knowledge Distillation on Graphs: A Surveycited by 38
  24. 2022Characterizing Datapoints via Second-Split Forgettingcited by 37
  25. 2022Efficient Split-Mix Federated Learning for On-Demand and In-Situ Customizationcited by 35
  26. 2023Understanding and Improving Feature Learning for Out-of-Distribution Generalizationcited by 35
  27. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networkscited by 35
  28. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  29. 2022On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Modelscited by 34
  30. 2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
  31. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillationcited by 31
  32. 2023Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasingcited by 31
  33. 2021Bagging, optimized dynamic mode decomposition (BOP-DMD) for robust, stable forecasting with spatial and temporal uncertainty-quantificationcited by 30
  34. 2023What Matters In The Structured Pruning of Generative Language Models?cited by 30
  35. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  36. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 28
  37. 2022Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learningcited by 26
  38. 2023Spurious Feature Diversification Improves Out-of-distribution Generalizationcited by 26

50 most cited of 297 All 297 (JSON)