# Prime-Aware Adaptive Distillation

arXiv: https://arxiv.org/abs/2008.01458 · HTML: https://mlsanity.com/paper/2008.01458/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Aug 2020.

## Built on by (most cited first, 28 of 28)

- [Distilling Object Detectors via Decoupled Features](https://mlsanity.com/paper/2103.14475/) (2021)
- [Data-Free Knowledge Transfer: A Survey](https://mlsanity.com/paper/2112.15278/) (2021)
- [Understanding the Role of the Projector in Knowledge Distillation](https://mlsanity.com/paper/2303.11098/) (2023)
- [Improved Feature Distillation via Projector Ensemble](https://mlsanity.com/paper/2210.15274/) (2022)
- [Teacher's pet: understanding and mitigating biases in distillation](https://mlsanity.com/paper/2106.10494/) (2021)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [LGD: Label-guided Self-distillation for Object Detection](https://mlsanity.com/paper/2109.11496/) (2021)
- [Information Theoretic Representation Distillation](https://mlsanity.com/paper/2112.00459/) (2021)
- [Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs](https://mlsanity.com/paper/2503.16870/) (2025)
- [Dynamic Contrastive Distillation for Image-Text Retrieval](https://mlsanity.com/paper/2207.01426/) (2022)
- [Generalized Knowledge Distillation via Relationship Matching](https://mlsanity.com/paper/2205.01915/) (2022)
- [EA-KD: Entropy-based Adaptive Knowledge Distillation](https://mlsanity.com/paper/2311.13621/) (2023)
- [LIX: Implicitly Infusing Spatial Geometric Prior Knowledge into Visual Semantic Segmentation for Autonomous Driving](https://mlsanity.com/paper/2403.08215/) (2024)
- [Avatar Knowledge Distillation: Self-ensemble Teacher Paradigm with Uncertainty](https://mlsanity.com/paper/2305.02722/) (2023)
- [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025)
- [Distilling Object Detectors With Global Knowledge](https://mlsanity.com/paper/2210.09022/) (2022)
- [Adaptive Instance Distillation for Object Detection in Autonomous Driving](https://mlsanity.com/paper/2201.11097/) (2022)
- [CPFD: Confidence-aware Privileged Feature Distillation for Short Video Classification](https://mlsanity.com/paper/2410.03038/) (2024)
- [Head3D: Complete 3D Head Generation via Tri-plane Feature Distillation](https://mlsanity.com/paper/2303.15892/) (2023)
- [Knowledge Distillation from Single to Multi Labels: an Empirical Study](https://mlsanity.com/paper/2303.08360/) (2023)
- [torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP](https://mlsanity.com/paper/2310.17644/) (2023)
- [Online Policy Distillation with Decision-Attention](https://mlsanity.com/paper/2406.05488/) (2024)
- [Understanding the Effects of Projectors in Knowledge Distillation](https://mlsanity.com/paper/2310.17183/) (2023)
- [Teaching with Uncertainty: Unleashing the Potential of Knowledge Distillation in Object Detection](https://mlsanity.com/paper/2406.06999/) (2024)
- [Structural and Statistical Audio Texture Knowledge Distillation for Acoustic Classification](https://mlsanity.com/paper/2501.01921/) (2025)
- [Shaping Parameter Contribution Patterns for Out-of-Distribution Detection](https://mlsanity.com/paper/2603.07195/) (2026)
- [NeuroGuard: Neural Gradient Update Aware of Representation Damage](https://mlsanity.com/paper/2608.08068/) (2026)
- [A Statistical Perspective on Knowledge Distillation: Foundations, Classical Methods, and Large Language Model Extensions](https://mlsanity.com/paper/2609.33727/) (2026)

## Abstract (arXiv)

Knowledge distillation(KD) aims to improve the performance of a student network by mimicing the knowledge from a powerful teacher network. Existing methods focus on studying what knowledge should be transferred and treat all samples equally during training. This paper introduces the adaptive sample weighting to KD. We discover that previous effective hard mining methods are not appropriate for distillation. Furthermore, we propose Prime-Aware Adaptive Distillation (PAD) by the incorporation of uncertainty learning. PAD perceives the prime samples in distillation and then emphasizes their effect adaptively. PAD is fundamentally different from and would refine existing methods with the innovative view of unequal training. For this reason, PAD is versatile and has been applied in various tasks including classification, metric learning, and object detection. With ten teacher-student combinations on six datasets, PAD promotes the performance of existing distillation methods and outperforms recent state-of-the-art methods.