Weight Averaging Improves Knowledge Distillation under Domain Shift
On the map: Knowledge distillation → General knowledge distillation methods
Abstract (arXiv)
Knowledge distillation (KD) is a powerful model compression technique broadly used in practical deep learning applications. It is focused on training a small student network to mimic a larger teacher network. While it is widely known that KD can offer an improvement to student generalization in i.i.d setting, its performance under domain shift, i.e. the performance of student networks on data from domains unseen during training, has received little attention in the literature. In this paper we make a step towards bridging the research fields of knowledge distillation and domain generalization. We show that weight averaging techniques proposed in domain generalization literature, such as SWAD and SMA, also improve the performance of knowledge distillation under domain shift. In addition, we propose a simplistic weight averaging strategy that does not require evaluation on validation data during training and show that it performs on par with SWAD and SMA when applied to KD. We name our final distillation approach Weight-Averaged Knowledge Distillation (WAKD).
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 44 of its 51 references that are in the map. First posted on arXiv 20 Sep 2023.
Builds on
44 of 51 references.
Closest ancestors · 7
Same direction, newest first
- 2023Robust Representation Learning with Self-Distillation for Domain Generalization14 Feb
- 2022On-Device Domain Generalization15 Sep
- 2022What Knowledge Gets Distilled in Knowledge Distillation?31 May
- 2022Decoupled Knowledge Distillation16 Mar
- 2021Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data27 Oct
- 2021Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation6 Jul
- 2021Knowledge distillation: A good teacher is patient and consistent9 Jun
Other references · 37
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
- 2014Going Deeper with Convolutionscited by 4,241
25 more
- 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
- 2014FitNets: Hints for Thin Deep Netscited by 1,653
- 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
- 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,188
- 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
- 2020In Search of Lost Domain Generalizationcited by 1,054
- 2017Deeper, Broader and Artier Domain Generalizationcited by 986
- 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
- 2021Domain Generalization: A Surveycited by 764
- 2017Learning to Generalize: Meta-Learning for Domain Generalizationcited by 763
- 2019Contrastive Representation Distillationcited by 748
- 2013Domain Generalization via Invariant Feature Representationcited by 623
- 2019Domain Generalization by Solving Jigsaw Puzzlescited by 450
- 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 434
- 2017Unified Deep Supervised Domain Adaptation and Generalizationcited by 323
- 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
- 2020Robust and Generalizable Visual Representation Learning via Random Convolutionscited by 142
- 2020Learning to Generalize Unseen Domains via Memory-based Multi-Source Meta-Learning for Person Re-Identificationcited by 110
- 2020Learning from Extrinsic and Intrinsic Supervisions for Domain Generalizationcited by 108
- 2020Towards Universal Representation Learning for Deep Face Recognitioncited by 78
- 2020Self-Supervised Learning Across Domainscited by 44
- 2022Self-Distilled Vision Transformer for Domain Generalizationcited by 35
- 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33
- 2020Improving out-of-distribution generalization via multi-task self-supervised pretrainingcited by 26
- 2022Cross-Domain Ensemble Distillation for Domain Generalizationcited by 21
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023RdimKD: Generic Distillation Paradigm by Dimensionality Reduction
- 2023Improved knowledge distillation by utilizing backward pass knowledge in neural networks
- 2024Efficient and Robust Knowledge Distillation from A Stronger Teacher Based on Correlation Matching
- 2023Generalization Matters: Loss Minima Flattening via Parameter Hybridization for Efficient Online Knowledge Distillation
- 2023What is Lost in Knowledge Distillation?
- 2025The Role of Teacher Calibration in Knowledge Distillation
- 2025UHKD: A Unified Framework for Heterogeneous Knowledge Distillation via Frequency-Domain Representations
- 2025Delta Knowledge Distillation for Large Language Models