# How Knowledge Distillation Mitigates the Synthetic Gap in Fair Face Recognition

arXiv: https://arxiv.org/abs/2408.17399 · HTML: https://mlsanity.com/paper/2408.17399/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 30 Aug 2024.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [AdaFace: Quality Adaptive Margin for Face Recognition](https://mlsanity.com/paper/2204.00964/) (2022)
- [Cross-Age LFW: A Database for Studying Cross-Age Face Recognition in Unconstrained Environments](https://mlsanity.com/paper/1708.08197/) (2017)
- [MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices](https://mlsanity.com/paper/1804.07573/) (2018)
- [ElasticFace: Elastic Margin Loss for Deep Face Recognition](https://mlsanity.com/paper/2109.09416/) (2021)
- [DCFace: Synthetic Face Generation with Dual Condition Diffusion Model](https://mlsanity.com/paper/2304.07060/) (2023)
- [DigiFace-1M: 1 Million Digital Face Images for Face Recognition](https://mlsanity.com/paper/2210.02579/) (2022)
- [IDiff-Face: Synthetic-based Face Recognition through Fizzy Identity-Conditioned Diffusion Models](https://mlsanity.com/paper/2308.04995/) (2023)
- [SFace: Privacy-friendly and Accurate Face Recognition using Synthetic Data](https://mlsanity.com/paper/2206.10520/) (2022)
- [Fair Feature Distillation for Visual Recognition](https://mlsanity.com/paper/2106.04411/) (2021)
- [Meta Balanced Network for Fair Face Recognition](https://mlsanity.com/paper/2205.06548/) (2022)
- [Synthetic Data for Face Recognition: Current State and Future Prospects](https://mlsanity.com/paper/2305.01021/) (2023)
- [PocketNet: Extreme Lightweight Face Recognition Network using Neural Architecture Search and Multi-Step Knowledge Distillation](https://mlsanity.com/paper/2108.10710/) (2021)
- [ExFaceGAN: Exploring Identity Directions in GAN's Learned Latent Space for Synthetic Identity Generation](https://mlsanity.com/paper/2307.05151/) (2023)
- [Second Edition FRCSyn Challenge at CVPR 2024: Face Recognition Challenge in the Era of Synthetic Data](https://mlsanity.com/paper/2404.10378/) (2024)
- [Evaluation-oriented Knowledge Distillation for Deep Face Recognition](https://mlsanity.com/paper/2206.02325/) (2022)
- [Compressed Models Decompress Race Biases: What Quantized Models Forget for Fair Face Recognition](https://mlsanity.com/paper/2308.11840/) (2023)
- [SynthDistill: Face Recognition with Knowledge Distillation from Synthetic Data](https://mlsanity.com/paper/2308.14852/) (2023)
- [AdaDistill: Adaptive Knowledge Distillation for Deep Face Recognition](https://mlsanity.com/paper/2407.01332/) (2024)
- [Model Compression Techniques in Biometrics Applications: A Survey](https://mlsanity.com/paper/2401.10139/) (2024)
- [If It's Not Enough, Make It So: Reducing Authentic Data Demand in Face Recognition through Synthetic Faces](https://mlsanity.com/paper/2404.03537/) (2024)
- [AI-KD: Towards Alignment Invariant Face Image Quality Assessment Using Knowledge Distillation](https://mlsanity.com/paper/2404.09555/) (2024)

## Built on by (most cited first, 1 of 1)

- [Balancing Beyond Discrete Categories: Continuous Demographic Labels for Fair Face Recognition](https://mlsanity.com/paper/2506.01532/) (2025)

## Abstract (arXiv)

Leveraging the capabilities of Knowledge Distillation (KD) strategies, we devise a strategy to fight the recent retraction of face recognition datasets. Given a pretrained Teacher model trained on a real dataset, we show that carefully utilising synthetic datasets, or a mix between real and synthetic datasets to distil knowledge from this teacher to smaller students can yield surprising results. In this sense, we trained 33 different models with and without KD, on different datasets, with different architectures and losses. And our findings are consistent, using KD leads to performance gains across all ethnicities and decreased bias. In addition, it helps to mitigate the performance gap between real and synthetic datasets. This approach addresses the limitations of synthetic data training, improving both the accuracy and fairness of face recognition models.