# Evaluation-oriented Knowledge Distillation for Deep Face Recognition

arXiv: https://arxiv.org/abs/2206.02325 · HTML: https://mlsanity.com/paper/2206.02325/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 6 Jun 2022.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 22 of 22)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognition](https://mlsanity.com/paper/2004.00288/) (2020)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [Cross-Age LFW: A Database for Studying Cross-Age Face Recognition in Unconstrained Environments](https://mlsanity.com/paper/1708.08197/) (2017)
- [The MegaFace Benchmark: 1 Million Faces for Recognition at Scale](https://mlsanity.com/paper/1512.00596/) (2015)
- [MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices](https://mlsanity.com/paper/1804.07573/) (2018)
- [Deep Learning Face Representation by Joint Identification-Verification](https://mlsanity.com/paper/1406.4773/) (2014)
- [Gradient Harmonized Single-stage Detector](https://mlsanity.com/paper/1811.05181/) (2018)
- [Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval](https://mlsanity.com/paper/2007.12163/) (2020)
- [DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer](https://mlsanity.com/paper/1707.01220/) (2017)
- [Triplet Distillation for Deep Face Recognition](https://mlsanity.com/paper/1905.04457/) (2019)
- [ShrinkTeaNet: Million-scale Lightweight Face Recognition via Shrinking Teacher-Student Networks](https://mlsanity.com/paper/1905.10620/) (2019)
- [MarginDistillation: distillation for margin-based softmax](https://mlsanity.com/paper/2003.02586/) (2020)

## Built on by (most cited first, 19 of 19)

- [Cross Architecture Distillation for Face Recognition](https://mlsanity.com/paper/2306.14662/) (2023)
- [AdaDistill: Adaptive Knowledge Distillation for Deep Face Recognition](https://mlsanity.com/paper/2407.01332/) (2024)
- [Foreground Object Search by Distilling Composite Image Feature](https://mlsanity.com/paper/2308.04990/) (2023)
- [VRM: Knowledge Distillation via Virtual Relation Matching](https://mlsanity.com/paper/2502.20760/) (2025)
- [Grouped Knowledge Distillation for Deep Face Recognition](https://mlsanity.com/paper/2304.04462/) (2023)
- [Model Compression Techniques in Biometrics Applications: A Survey](https://mlsanity.com/paper/2401.10139/) (2024)
- [Cross-View Consistency Regularisation for Knowledge Distillation](https://mlsanity.com/paper/2412.16493/) (2024)
- [Cross-Architecture Distillation Made Simple with Redundancy Suppression](https://mlsanity.com/paper/2507.21844/) (2025)
- [Attention-guided Feature Distillation for Semantic Segmentation](https://mlsanity.com/paper/2403.05451/) (2024)
- [Descriptor Distillation: a Teacher-Student-Regularized Framework for Learning Local Descriptors](https://mlsanity.com/paper/2209.11795/) (2022)
- [AICSD: Adaptive Inter-Class Similarity Distillation for Semantic Segmentation](https://mlsanity.com/paper/2308.04243/) (2023)
- [MST-KD: Multiple Specialized Teachers Knowledge Distillation for Fair Face Recognition](https://mlsanity.com/paper/2408.16563/) (2024)
- [Sample Correlation for Fingerprinting Deep Face Recognition](https://mlsanity.com/paper/2412.20768/) (2024)
- [QCFace: Image Quality Control for boosting Face Representation & Recognition](https://mlsanity.com/paper/2510.15289/) (2025)
- [On Biased Behavior of GANs for Face Verification](https://mlsanity.com/paper/2208.13061/) (2022)
- [A Survey of Face Recognition](https://mlsanity.com/paper/2212.13038/) (2022)
- [How Knowledge Distillation Mitigates the Synthetic Gap in Fair Face Recognition](https://mlsanity.com/paper/2408.17399/) (2024)
- [Ef-QuantFace: Streamlined Face Recognition with Small Data and Low-Bit Precision](https://mlsanity.com/paper/2402.18163/) (2024)
- [A Layered Self-Supervised Knowledge Distillation Framework for Efficient Multimodal Learning on the Edge](https://mlsanity.com/paper/2506.07055/) (2025)

## Abstract (arXiv)

Knowledge distillation (KD) is a widely-used technique that utilizes large networks to improve the performance of compact models. Previous KD approaches usually aim to guide the student to mimic the teacher's behavior completely in the representation space. However, such one-to-one corresponding constraints may lead to inflexible knowledge transfer from the teacher to the student, especially those with low model capacities. Inspired by the ultimate goal of KD methods, we propose a novel Evaluation oriented KD method (EKD) for deep face recognition to directly reduce the performance gap between the teacher and student models during training. Specifically, we adopt the commonly used evaluation metrics in face recognition, i.e., False Positive Rate (FPR) and True Positive Rate (TPR) as the performance indicator. According to the evaluation protocol, the critical pair relations that cause the TPR and FPR difference between the teacher and student models are selected. Then, the critical relations in the student are constrained to approximate the corresponding ones in the teacher by a novel rank-based loss function, giving more flexibility to the student with low capacity. Extensive experimental results on popular benchmarks demonstrate the superiority of our EKD over state-of-the-art competitors.