# What Makes a "Good" Data Augmentation in Knowledge Distillation -- A Statistical Perspective

arXiv: https://arxiv.org/abs/2012.02909 · HTML: https://mlsanity.com/paper/2012.02909/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2020.

## Built on by (most cited first, 30 of 35)

- [Binarized Spectral Compressive Imaging](https://mlsanity.com/paper/2305.10299/) (2023)
- [Binarized Diffusion Model for Image Super-Resolution](https://mlsanity.com/paper/2406.05723/) (2024)
- [Binarized Low-light Raw Video Enhancement](https://mlsanity.com/paper/2403.19944/) (2024)
- [On student-teacher deviations in distillation: does it pay to disobey?](https://mlsanity.com/paper/2301.12923/) (2023)
- [Towards Distribution-Agnostic Generalized Category Discovery](https://mlsanity.com/paper/2310.01376/) (2023)
- [VanillaKD: Revisit the Power of Vanilla Knowledge Distillation from Small Scale to Large Scale](https://mlsanity.com/paper/2305.15781/) (2023)
- [Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models](https://mlsanity.com/paper/2311.01441/) (2023)
- [Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learning](https://mlsanity.com/paper/2305.16379/) (2023)
- [Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clustering](https://mlsanity.com/paper/2307.11030/) (2023)
- [Not All LLM-Generated Data Are Equal: Rethinking Data Weighting in Text Classification](https://mlsanity.com/paper/2410.21526/) (2024)
- [Cross-View Consistency Regularisation for Knowledge Distillation](https://mlsanity.com/paper/2412.16493/) (2024)
- [Robustness-Guided Image Synthesis for Data-Free Quantization](https://mlsanity.com/paper/2310.03661/) (2023)
- [Comparative Knowledge Distillation](https://mlsanity.com/paper/2311.02253/) (2023)
- [Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning](https://mlsanity.com/paper/2311.13934/) (2023)
- [Revealing the Two Sides of Data Augmentation: An Asymmetric Distillation-based Win-Win Solution for Open-Set Recognition](https://mlsanity.com/paper/2404.19527/) (2024)
- [DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer](https://mlsanity.com/paper/2505.15133/) (2025)
- [Maximizing Discrimination Capability of Knowledge Distillation with Energy Function](https://mlsanity.com/paper/2311.14334/) (2023)
- [On Good Practices for Task-Specific Distillation of Large Pretrained Visual Models](https://mlsanity.com/paper/2402.11305/) (2024)
- [Overcoming Uncertain Incompleteness for Robust Multimodal Sequential Diagnosis Prediction via Curriculum Data Erasing Guided Knowledge Distillation](https://mlsanity.com/paper/2407.19540/) (2024)
- [Distilling Calibrated Student from an Uncalibrated Teacher](https://mlsanity.com/paper/2302.11472/) (2023)
- [Understanding the Effect of Data Augmentation on Knowledge Distillation](https://mlsanity.com/paper/2305.12565/) (2023)
- [ShiftKD: Benchmarking Knowledge Distillation under Distribution Shift](https://mlsanity.com/paper/2312.16242/) (2023)
- [Improving Knowledge Distillation Under Unknown Covariate Shift Through Confidence-Guided Data Augmentation](https://mlsanity.com/paper/2506.02294/) (2025)
- [Precise Knowledge Transfer via Flow Matching](https://mlsanity.com/paper/2402.02012/) (2024)
- [Weights Augmentation: it has never ever ever ever let her model down](https://mlsanity.com/paper/2405.19590/) (2024)
- [Diffusion-based Data Augmentation and Knowledge Distillation with Generated Soft Labels Solving Data Scarcity Problems of SAR Oil Spill Segmentation](https://mlsanity.com/paper/2412.08116/) (2024)
- [Real-Time Cell Sorting with Scalable In Situ FPGA-Accelerated Deep Learning](https://mlsanity.com/paper/2503.12622/) (2025)
- [Data-Augmented Quantization-Aware Knowledge Distillation](https://mlsanity.com/paper/2509.03850/) (2025)
- [QVGGT: Post-Training Quantized Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2605.31124/) (2026)
- [DualMix: Unleashing the Potential of Data Augmentation for Online Class-Incremental Learning](https://mlsanity.com/paper/2303.07864/) (2023)

## Abstract (arXiv)

Knowledge distillation (KD) is a general neural network training approach that uses a teacher model to guide the student model. Existing works mainly study KD from the network output side (e.g., trying to design a better KD loss function), while few have attempted to understand it from the input side. Especially, its interplay with data augmentation (DA) has not been well understood. In this paper, we ask: Why do some DA schemes (e.g., CutMix) inherently perform much better than others in KD? What makes a "good" DA in KD? Our investigation from a statistical perspective suggests that a good DA scheme should reduce the covariance of the teacher-student cross-entropy. A practical metric, the stddev of teacher's mean probability (T. stddev), is further presented and well justified empirically. Besides the theoretical understanding, we also introduce a new entropy-based data-mixing DA scheme, CutMixPick, to further enhance CutMix. Extensive empirical studies support our claims and demonstrate how we can harvest considerable performance gains simply by using a better DA scheme in knowledge distillation.