# Knowledge distillation via adaptive instance normalization

arXiv: https://arxiv.org/abs/2003.04289 · HTML: https://mlsanity.com/paper/2003.04289/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2020.

## Built on by (most cited first, 14 of 14)

- [A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance](https://mlsanity.com/paper/2309.12530/) (2023)
- [Visual Tuning](https://mlsanity.com/paper/2305.06061/) (2023)
- [CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks](https://mlsanity.com/paper/2201.05729/) (2022)
- [Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Tasks](https://mlsanity.com/paper/2204.10496/) (2022)
- [Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillation](https://mlsanity.com/paper/2112.03473/) (2021)
- [Coaching a Teachable Student](https://mlsanity.com/paper/2306.10014/) (2023)
- [Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation](https://mlsanity.com/paper/2412.08139/) (2024)
- [Direct Distillation between Different Domains](https://mlsanity.com/paper/2401.06826/) (2024)
- [VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognition](https://mlsanity.com/paper/2408.16930/) (2024)
- [Compacting Deep Neural Networks for Internet of Things: Methods and Applications](https://mlsanity.com/paper/2103.11083/) (2021)
- [FSNet: Redesign Self-Supervised MonoDepth for Full-Scale Depth Prediction for Autonomous Driving](https://mlsanity.com/paper/2304.10719/) (2023)
- [Scalable Vision-Based 3D Object Detection and Monocular Depth Estimation for Autonomous Driving](https://mlsanity.com/paper/2403.02037/) (2024)
- [A Transformer-in-Transformer Network Utilizing Knowledge Distillation for Image Recognition](https://mlsanity.com/paper/2502.16762/) (2025)
- [Vision Transformer for Robust Occluded Person Reidentification in Complex Surveillance Scenes](https://mlsanity.com/paper/2510.27677/) (2025)

## Abstract (arXiv)

This paper addresses the problem of model compression via knowledge distillation. To this end, we propose a new knowledge distillation method based on transferring feature statistics, specifically the channel-wise mean and variance, from the teacher to the student. Our method goes beyond the standard way of enforcing the mean and variance of the student to be similar to those of the teacher through an L2 loss, which we found it to be of limited effectiveness. Specifically, we propose a new loss based on adaptive instance normalization to effectively transfer the feature statistics. The main idea is to transfer the learned statistics back to the teacher via adaptive instance normalization (conditioned on the student) and let the teacher network "evaluate" via a loss whether the statistics learned by the student are reliably transferred. We show that our distillation method outperforms other state-of-the-art distillation methods over a large set of experimental settings including different (a) network architectures, (b) teacher-student capacities, (c) datasets, and (d) domains.