# SynthDistill: Face Recognition with Knowledge Distillation from Synthetic Data arXiv: https://arxiv.org/abs/2308.14852 ยท HTML: https://mlsanity.com/paper/2308.14852/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. - Builds on 47 of its 58 references that are in the map. - First posted on arXiv 28 Aug 2023. ## Placed in - Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/) ## Builds on (most cited first, 30 of 47) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019) - [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018) - [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017) - [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014) - [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020) - [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018) - [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016) - [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018) - [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017) - [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019) - [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016) - [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019) - [Learning Face Representation from Scratch](https://mlsanity.com/paper/1411.7923/) (2014) - [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019) - [AdaFace: Quality Adaptive Margin for Face Recognition](https://mlsanity.com/paper/2204.00964/) (2022) - [Distilling Knowledge via Knowledge Review](https://mlsanity.com/paper/2104.09044/) (2021) - [Cross-Age LFW: A Database for Studying Cross-Age Face Recognition in Unconstrained Environments](https://mlsanity.com/paper/1708.08197/) (2017) - [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018) - [WebFace260M: A Benchmark Unveiling the Power of Million-Scale Deep Face Recognition](https://mlsanity.com/paper/2103.04098/) (2021) - [Disentangled and Controllable Face Image Generation via 3D Imitative-Contrastive Learning](https://mlsanity.com/paper/2004.11660/) (2020) - [Data-Free Learning of Student Networks](https://mlsanity.com/paper/1904.01186/) (2019) - [Knowledge Distillation by On-the-Fly Native Ensemble](https://mlsanity.com/paper/1806.04606/) (2018) - [MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices](https://mlsanity.com/paper/1804.07573/) (2018) - [Data-Free Knowledge Distillation for Deep Neural Networks](https://mlsanity.com/paper/1710.07535/) (2017) - [Online Knowledge Distillation with Diverse Peers](https://mlsanity.com/paper/1912.00350/) (2019) - [A Light CNN for Deep Face Representation with Noisy Labels](https://mlsanity.com/paper/1511.02683/) (2015) - [DCFace: Synthetic Face Generation with Dual Condition Diffusion Model](https://mlsanity.com/paper/2304.07060/) (2023) - [Curriculum Temperature for Knowledge Distillation](https://mlsanity.com/paper/2211.16231/) (2022) ## Built on by (most cited first, 9 of 9) - [SDFR: Synthetic Data for Face Recognition Competition](https://mlsanity.com/paper/2404.04580/) (2024) - [Second FRCSyn-onGoing: Winning Solutions and Post-Challenge Analysis to Improve Face Recognition with Synthetic Data](https://mlsanity.com/paper/2412.01383/) (2024) - [Digi2Real: Bridging the Realism Gap in Synthetic Data Face Recognition via Foundation Models](https://mlsanity.com/paper/2411.02188/) (2024) - [Unveiling Synthetic Faces: How Synthetic Datasets Can Expose Real Identities](https://mlsanity.com/paper/2410.24015/) (2024) - [MST-KD: Multiple Specialized Teachers Knowledge Distillation for Fair Face Recognition](https://mlsanity.com/paper/2408.16563/) (2024) - [How Knowledge Distillation Mitigates the Synthetic Gap in Fair Face Recognition](https://mlsanity.com/paper/2408.17399/) (2024) - [Enhancing Domain Diversity in Synthetic Data Face Recognition with Dataset Fusion](https://mlsanity.com/paper/2507.16790/) (2025) - [Quality-Aware Multimodal Fusion Reveals Implicit Identity in Valence-Arousal Features](https://mlsanity.com/paper/2607.21347/) (2026) - [SynThermFace: Amplifying Limited Paired Data for Visible-Thermal Face Recognition via Synthetic Data Generation](https://mlsanity.com/paper/2609.10303/) (2026) ## Abstract (arXiv) State-of-the-art face recognition networks are often computationally expensive and cannot be used for mobile applications. Training lightweight face recognition models also requires large identity-labeled datasets. Meanwhile, there are privacy and ethical concerns with collecting and using large face recognition datasets. While generating synthetic datasets for training face recognition models is an alternative option, it is challenging to generate synthetic data with sufficient intra-class variations. In addition, there is still a considerable gap between the performance of models trained on real and synthetic data. In this paper, we propose a new framework (named SynthDistill) to train lightweight face recognition models by distilling the knowledge of a pretrained teacher face recognition model using synthetic data. We use a pretrained face generator network to generate synthetic face images and use the synthesized images to learn a lightweight student network. We use synthetic face images without identity labels, mitigating the problems in the intra-class variation generation of synthetic datasets. Instead, we propose a novel dynamic sampling strategy from the intermediate latent space of the face generator network to include new variations of the challenging images while further exploring new face images in the training batch. The results on five different face recognition datasets demonstrate the superiority of our lightweight model compared to models trained on previous synthetic datasets, achieving a verification accuracy of 99.52% on the LFW dataset with a lightweight network. The results also show that our proposed framework significantly reduces the gap between training with real and synthetic data. The source code for replicating the experiments is publicly released.