# Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation

arXiv: https://arxiv.org/abs/2107.02629 · HTML: https://mlsanity.com/paper/2107.02629/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 44 of its 67 references that are in the map.
- First posted on arXiv 6 Jul 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 44)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [Generalizing to Unseen Domains via Adversarial Data Augmentation](https://mlsanity.com/paper/1805.12018/) (2018)
- [Domain Generalization by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1903.06864/) (2019)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Self-Challenging Improves Cross-Domain Generalization](https://mlsanity.com/paper/2007.02454/) (2020)
- [Domain Generalization via Model-Agnostic Learning of Semantic Features](https://mlsanity.com/paper/1910.13580/) (2019)
- [Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks](https://mlsanity.com/paper/1612.05424/) (2016)
- [A Probabilistic U-Net for Segmentation of Ambiguous Images](https://mlsanity.com/paper/1806.05034/) (2018)
- [Generalizing Across Domains via Cross-Gradient Training](https://mlsanity.com/paper/1804.10745/) (2018)
- [Domain Generalization for Object Recognition with Multi-task Autoencoders](https://mlsanity.com/paper/1508.07680/) (2015)
- [Episodic Training for Domain Generalization](https://mlsanity.com/paper/1902.00113/) (2019)
- [Unifying distillation and privileged information](https://mlsanity.com/paper/1511.03643/) (2015)
- [Deep Domain-Adversarial Image Generation for Domain Generalisation](https://mlsanity.com/paper/2003.06054/) (2020)
- [Invariant Risk Minimization Games](https://mlsanity.com/paper/2002.04692/) (2020)

## Built on by (most cited first, 29 of 29)

- [Domain Generalization for Medical Image Analysis: A Review](https://mlsanity.com/paper/2310.08598/) (2023)
- [A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance](https://mlsanity.com/paper/2309.12530/) (2023)
- [Self-Distilled Vision Transformer for Domain Generalization](https://mlsanity.com/paper/2207.12392/) (2022)
- [Part-Aware Transformer for Generalizable Person Re-identification](https://mlsanity.com/paper/2308.03322/) (2023)
- [DiGA: Distil to Generalize and then Adapt for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2304.02222/) (2023)
- [Variational Disentanglement for Domain Generalization](https://mlsanity.com/paper/2109.05826/) (2021)
- [Label-Efficient Domain Generalization via Collaborative Exploration and Generalization](https://mlsanity.com/paper/2208.03644/) (2022)
- [PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization](https://mlsanity.com/paper/2404.09011/) (2024)
- [TFS-ViT: Token-Level Feature Stylization for Domain Generalization](https://mlsanity.com/paper/2303.15698/) (2023)
- [Diverse Target and Contribution Scheduling for Domain Generalization](https://mlsanity.com/paper/2309.16460/) (2023)
- [Can LLMs Find Fraudsters? Multi-level LLM Enhanced Graph Fraud Detection](https://mlsanity.com/paper/2507.11997/) (2025)
- [Generalizing to Unseen Domains in Diabetic Retinopathy Classification](https://mlsanity.com/paper/2310.17255/) (2023)
- [Unleashing the Potential of Pre-Trained Diffusion Models for Generalizable Person Re-Identification](https://mlsanity.com/paper/2502.06619/) (2025)
- [Random Teachers are Good Teachers](https://mlsanity.com/paper/2302.12091/) (2023)
- [Domain Generalization with Small Data](https://mlsanity.com/paper/2402.06150/) (2024)
- [Domain Generalization for Crop Segmentation with Standardized Ensemble Knowledge Distillation](https://mlsanity.com/paper/2304.01029/) (2023)
- [FDS: Feedback-guided Domain Synthesis with Multi-Source Conditional Diffusion Models for Domain Generalization](https://mlsanity.com/paper/2407.03588/) (2024)
- [PLACE dropout: A Progressive Layer-wise and Channel-wise Dropout for Domain Generalization](https://mlsanity.com/paper/2112.03676/) (2021)
- [Exploring Generalizable Distillation for Efficient Medical Image Segmentation](https://mlsanity.com/paper/2207.12995/) (2022)
- [Removing Image Artifacts From Scratched Lens Protectors](https://mlsanity.com/paper/2302.05746/) (2023)
- [Towards domain generalisation in ASR with elitist sampling and ensemble knowledge distillation](https://mlsanity.com/paper/2303.00550/) (2023)
- [ShiftKD: Benchmarking Knowledge Distillation under Distribution Shift](https://mlsanity.com/paper/2312.16242/) (2023)
- [Human Heterogeneity Invariant Stress Sensing](https://mlsanity.com/paper/2506.02256/) (2025)
- [FEDTAIL: Federated Long-Tailed Domain Generalization with Sharpness-Guided Gradient Matching](https://mlsanity.com/paper/2506.08518/) (2025)
- [Cross-domain Transfer of defect features in technical domains based on partial target data](https://mlsanity.com/paper/2211.13662/) (2022)
- [Weight Averaging Improves Knowledge Distillation under Domain Shift](https://mlsanity.com/paper/2309.11446/) (2023)
- [MUST: A Multilingual Student-Teacher Learning approach for low-resource speech recognition](https://mlsanity.com/paper/2310.18865/) (2023)
- [All Centers Are at most a Few Tokens Apart: Knowledge Distillation with Domain Invariant Prompt Tuning](https://mlsanity.com/paper/2511.22739/) (2025)
- [ROBUST-WT: Robust Uncertainty-aware Segmentation Transform via Whitening and Training Enhancements](https://mlsanity.com/paper/2606.03069/) (2026)

## Abstract (arXiv)

Though convolutional neural networks are widely used in different tasks, lack of generalization capability in the absence of sufficient and representative data is one of the challenges that hinder their practical application. In this paper, we propose a simple, effective, and plug-and-play training strategy named Knowledge Distillation for Domain Generalization (KDDG) which is built upon a knowledge distillation framework with the gradient filter as a novel regularization term. We find that both the ``richer dark knowledge" from the teacher network, as well as the gradient filter we proposed, can reduce the difficulty of learning the mapping which further improves the generalization ability of the model. We also conduct experiments extensively to show that our framework can significantly improve the generalization capability of deep neural networks in different tasks including image classification, segmentation, reinforcement learning by comparing our method with existing state-of-the-art domain generalization techniques. Last but not the least, we propose to adopt two metrics to analyze our proposed method in order to better understand how our proposed method benefits the generalization capability of deep neural networks.