# Grouped Knowledge Distillation for Deep Face Recognition arXiv: https://arxiv.org/abs/2304.04462 ยท HTML: https://mlsanity.com/paper/2304.04462/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. - Builds on 26 of its 41 references that are in the map. - First posted on arXiv 10 Apr 2023. ## Placed in - Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/) ## Builds on (most cited first, 26 of 26) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015) - [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019) - [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014) - [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018) - [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019) - [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017) - [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017) - [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019) - [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019) - [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014) - [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022) - [CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognition](https://mlsanity.com/paper/2004.00288/) (2020) - [Cross-Age LFW: A Database for Studying Cross-Age Face Recognition in Unconstrained Environments](https://mlsanity.com/paper/1708.08197/) (2017) - [The MegaFace Benchmark: 1 Million Faces for Recognition at Scale](https://mlsanity.com/paper/1512.00596/) (2015) - [MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices](https://mlsanity.com/paper/1804.07573/) (2018) - [Deep Learning Face Representation by Joint Identification-Verification](https://mlsanity.com/paper/1406.4773/) (2014) - [Nested Collaborative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2203.15359/) (2022) - [DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer](https://mlsanity.com/paper/1707.01220/) (2017) - [ArcFace: Additive Angular Margin Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.07698/) (2018) - [FaceX-Zoo: A PyTorch Toolbox for Face Recognition](https://mlsanity.com/paper/2101.04407/) (2021) - [Evaluation-oriented Knowledge Distillation for Deep Face Recognition](https://mlsanity.com/paper/2206.02325/) (2022) - [Triplet Distillation for Deep Face Recognition](https://mlsanity.com/paper/1905.04457/) (2019) - [ShrinkTeaNet: Million-scale Lightweight Face Recognition via Shrinking Teacher-Student Networks](https://mlsanity.com/paper/1905.10620/) (2019) - [Mask-invariant Face Recognition through Template-level Knowledge Distillation](https://mlsanity.com/paper/2112.05646/) (2021) - [MarginDistillation: distillation for margin-based softmax](https://mlsanity.com/paper/2003.02586/) (2020) ## Built on by (most cited first, 5 of 5) - [KeyPoint Relative Position Encoding for Face Recognition](https://mlsanity.com/paper/2403.14852/) (2024) - [Cross Architecture Distillation for Face Recognition](https://mlsanity.com/paper/2306.14662/) (2023) - [Unified Knowledge Distillation Framework: Fine-Grained Alignment and Geometric Relationship Preservation for Deep Face Recognition](https://mlsanity.com/paper/2508.11376/) (2025) - [Rethinking Decoupled Knowledge Distillation: A Predictive Distribution Perspective](https://mlsanity.com/paper/2512.04625/) (2025) - [Triage knowledge distillation for speaker verification](https://mlsanity.com/paper/2601.14699/) (2026) ## Abstract (arXiv) Compared with the feature-based distillation methods, logits distillation can liberalize the requirements of consistent feature dimension between teacher and student networks, while the performance is deemed inferior in face recognition. One major challenge is that the light-weight student network has difficulty fitting the target logits due to its low model capacity, which is attributed to the significant number of identities in face recognition. Therefore, we seek to probe the target logits to extract the primary knowledge related to face identity, and discard the others, to make the distillation more achievable for the student network. Specifically, there is a tail group with near-zero values in the prediction, containing minor knowledge for distillation. To provide a clear perspective of its impact, we first partition the logits into two groups, i.e., Primary Group and Secondary Group, according to the cumulative probability of the softened prediction. Then, we reorganize the Knowledge Distillation (KD) loss of grouped logits into three parts, i.e., Primary-KD, Secondary-KD, and Binary-KD. Primary-KD refers to distilling the primary knowledge from the teacher, Secondary-KD aims to refine minor knowledge but increases the difficulty of distillation, and Binary-KD ensures the consistency of knowledge distribution between teacher and student. We experimentally found that (1) Primary-KD and Binary-KD are indispensable for KD, and (2) Secondary-KD is the culprit restricting KD at the bottleneck. Therefore, we propose a Grouped Knowledge Distillation (GKD) that retains the Primary-KD and Binary-KD but omits Secondary-KD in the ultimate KD loss calculation. Extensive experimental results on popular face recognition benchmarks demonstrate the superiority of proposed GKD over state-of-the-art methods.