mlsanityData to 7 Oct 2026

Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation

arXiv 2206.15047 ↗First posted 30 Jun 2022cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Ensembles of deep neural networks have demonstrated superior performance, but their heavy computational cost hinders applying them for resource-limited environments. It motivates distilling knowledge from the ensemble teacher into a smaller student network, and there are two important design choices for this ensemble distillation: 1) how to construct the student network, and 2) what data should be shown during training. In this paper, we propose a weight averaging technique where a student with multiple subnetworks is trained to absorb the functional diversity of ensemble teachers, but then those subnetworks are properly averaged for inference, giving a single student network with no additional inference cost. We also propose a perturbation strategy that seeks inputs from which the diversities of teachers can be better transferred to the student. Combining these two, our method significantly improves upon previous methods on various image classification tasks.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 30 Jun 2022.

Builds on

25 of 33 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  7. 2016Wide Residual Networkscited by 3,057
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  9. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  10. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  11. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  12. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
13 more
  1. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  2. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 208
  3. 2019Ensemble Distribution Distillationcited by 158
  4. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 149
  5. 2021Repulsive Deep Ensembles are Bayesiancited by 103
  6. 2021Learning Neural Network Subspacescited by 82
  7. 2018Knowledge Transfer with Jacobian Matchingcited by 73
  8. 2019Knowledge Distillation via Route Constrained Optimizationcited by 73
  9. 2020Neural Ensemble Search for Uncertainty Estimation and Dataset Shiftcited by 56
  10. 2021DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimationcited by 48
  11. 2020Hydra: Preserving Ensemble Diversity for Model Distillationcited by 31
  12. 2021Diversity Matters When Learning From Ensemblescited by 27
  13. 2021Scaling Ensemble Distribution Distillation to Many Classes with Proxy Targetscited by 19

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4; 2024: 0; 2025: 1.

6 papers.

  1. 2023Deep Model Fusion: A Surveycited by 89
  2. 2022Accelerating Dataset Distillation via Model Augmentationcited by 62
  3. 2023Lookaround Optimizer: k steps around, 1 step averagecited by 9
  4. 2023AST: Effective Dataset Distillation through Alignment with Smooth and High-Quality Expert Trajectoriescited by 1
  5. 2025Knowledge Distillation of Uncertainty using Deep Latent Factor Modelcited by 1
  6. 2023Towards Efficient Deep Hashing Retrieval: Condensing Your Data via Feature-Embedding Matchingcited by 0