# AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation

arXiv: https://arxiv.org/abs/2403.07030 · HTML: https://mlsanity.com/paper/2403.07030/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 47 of its 66 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2002.08546/) (2020)
- [VisDA: The Visual Domain Adaptation Challenge](https://mlsanity.com/paper/1710.06924/) (2017)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Environment Inference for Invariant Learning](https://mlsanity.com/paper/2010.07249/) (2020)
- [Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation](https://mlsanity.com/paper/2110.04202/) (2021)
- [Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer](https://mlsanity.com/paper/2012.07297/) (2020)
- [Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Data](https://mlsanity.com/paper/2110.03374/) (2021)
- [Zero-shot Knowledge Transfer via Adversarial Belief Matching](https://mlsanity.com/paper/1905.09768/) (2019)
- [Data-Free Model Extraction](https://mlsanity.com/paper/2011.14779/) (2020)
- [Source-Free Domain Adaptation via Distribution Estimation](https://mlsanity.com/paper/2204.11257/) (2022)
- [Dynamic Curriculum Learning for Imbalanced Data Classification](https://mlsanity.com/paper/1901.06783/) (2019)
- [Data-Free Network Quantization With Adversarial Knowledge Distillation](https://mlsanity.com/paper/2005.04136/) (2020)
- [Stable Prediction across Unknown Environments](https://mlsanity.com/paper/1806.06270/) (2018)
- [C-SFDA: A Curriculum Learning Aided Self-Training Framework for Efficient Source Free Domain Adaptation](https://mlsanity.com/paper/2303.17132/) (2023)
- [Uncertainty-guided Source-free Domain Adaptation](https://mlsanity.com/paper/2208.07591/) (2022)
- [Prompt-based Distribution Alignment for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2312.09553/) (2023)
- [DeVLBert: Learning Deconfounded Visio-Linguistic Representations](https://mlsanity.com/paper/2008.06884/) (2020)
- [Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replay](https://mlsanity.com/paper/2201.03019/) (2022)

## Built on by (most cited first, 5 of 5)

- [What Makes a Good Dataset for Knowledge Distillation?](https://mlsanity.com/paper/2411.12817/) (2024)
- [Model Inversion with Layer-Specific Modeling and Alignment for Data-Free Continual Learning](https://mlsanity.com/paper/2510.26311/) (2025)
- [EvoDriveVLA: Evolving Driving VLA Models via Collaborative Perception-Planning Distillation](https://mlsanity.com/paper/2603.09465/) (2026)
- [Beyond Dark Knowledge: Mixup-Based Distillation for Reliable Predictions](https://mlsanity.com/paper/2606.12171/) (2026)
- [ConsistentFeature: A Plug-and-Play Component for Neural Network Regularization](https://mlsanity.com/paper/2412.01476/) (2024)

## Abstract (arXiv)

Due to privacy or patent concerns, a growing number of large models are released without granting access to their training data, making transferring their knowledge inefficient and problematic. In response, Data-Free Knowledge Distillation (DFKD) methods have emerged as direct solutions. However, simply adopting models derived from DFKD for real-world applications suffers significant performance degradation, due to the discrepancy between teachers' training data and real-world scenarios (student domain). The degradation stems from the portions of teachers' knowledge that are not applicable to the student domain. They are specific to the teacher domain and would undermine students' performance. Hence, selectively transferring teachers' appropriate knowledge becomes the primary challenge in DFKD. In this work, we propose a simple but effective method AuG-KD. It utilizes an uncertainty-guided and sample-specific anchor to align student-domain data with the teacher domain and leverages a generative method to progressively trade off the learning process between OOD knowledge distillation and domain-specific information learning via mixup learning. Extensive experiments in 3 datasets and 8 settings demonstrate the stability and superiority of our approach. Code available at https://github.com/IshiKura-a/AuG-KD .