# MST-KD: Multiple Specialized Teachers Knowledge Distillation for Fair Face Recognition

arXiv: https://arxiv.org/abs/2408.16563 · HTML: https://mlsanity.com/paper/2408.16563/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [Learning Face Representation from Scratch](https://mlsanity.com/paper/1411.7923/) (2014)
- [ElasticFace: Elastic Margin Loss for Deep Face Recognition](https://mlsanity.com/paper/2109.09416/) (2021)
- [Deep Imbalanced Learning for Face Recognition and Attribute Prediction](https://mlsanity.com/paper/1806.00194/) (2018)
- [Investigating Bias and Fairness in Facial Expression Recognition](https://mlsanity.com/paper/2007.10075/) (2020)
- [A Comprehensive Study on Face Recognition Biases Beyond Demographics](https://mlsanity.com/paper/2103.01592/) (2021)
- [Face Recognition: Too Bias, or Not Too Bias?](https://mlsanity.com/paper/2002.06483/) (2020)
- [Low-resolution Face Recognition in the Wild via Selective Knowledge Distillation](https://mlsanity.com/paper/1811.09998/) (2018)
- [Meta Balanced Network for Fair Face Recognition](https://mlsanity.com/paper/2205.06548/) (2022)
- [Confidence-Aware Multi-Teacher Knowledge Distillation](https://mlsanity.com/paper/2201.00007/) (2021)
- [How Does Gender Balance In Training Data Affect Face Recognition Accuracy?](https://mlsanity.com/paper/2002.02934/) (2020)
- [PocketNet: Extreme Lightweight Face Recognition Network using Neural Architecture Search and Multi-Step Knowledge Distillation](https://mlsanity.com/paper/2108.10710/) (2021)
- [Is Face Recognition Sexist? No, Gendered Hairstyles and Biology Are](https://mlsanity.com/paper/2008.06989/) (2020)
- [Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networks](https://mlsanity.com/paper/2210.16103/) (2022)
- [Evaluation-oriented Knowledge Distillation for Deep Face Recognition](https://mlsanity.com/paper/2206.02325/) (2022)
- [Longitudinal Study of Child Face Recognition](https://mlsanity.com/paper/1711.03990/) (2017)
- [Simon Says: Evaluating and Mitigating Bias in Pruned Neural Networks with Knowledge Distillation](https://mlsanity.com/paper/2106.07849/) (2021)
- [FocusFace: Multi-task Contrastive Learning for Masked Face Recognition](https://mlsanity.com/paper/2110.14940/) (2021)
- [Mask-invariant Face Recognition through Template-level Knowledge Distillation](https://mlsanity.com/paper/2112.05646/) (2021)
- [Causality-Inspired Taxonomy for Explainable Artificial Intelligence](https://mlsanity.com/paper/2208.09500/) (2022)
- [Unveiling the Two-Faced Truth: Disentangling Morphed Identities for Face Morphing Detection](https://mlsanity.com/paper/2306.03002/) (2023)
- [Compressed Models Decompress Race Biases: What Quantized Models Forget for Fair Face Recognition](https://mlsanity.com/paper/2308.11840/) (2023)
- [SynthDistill: Face Recognition with Knowledge Distillation from Synthetic Data](https://mlsanity.com/paper/2308.14852/) (2023)
- [CoupleFace: Relation Matters for Face Recognition Distillation](https://mlsanity.com/paper/2204.05502/) (2022)
- [Towards Explaining Demographic Bias through the Eyes of Face Recognition Models](https://mlsanity.com/paper/2208.13400/) (2022)
- [Model Compression Techniques in Biometrics Applications: A Survey](https://mlsanity.com/paper/2401.10139/) (2024)

## Built on by (most cited first, 2 of 2)

- [Review of Demographic Fairness in Face Recognition](https://mlsanity.com/paper/2502.02309/) (2025)
- [Balancing Beyond Discrete Categories: Continuous Demographic Labels for Fair Face Recognition](https://mlsanity.com/paper/2506.01532/) (2025)

## Abstract (arXiv)

As in school, one teacher to cover all subjects is insufficient to distill equally robust information to a student. Hence, each subject is taught by a highly specialised teacher. Following a similar philosophy, we propose a multiple specialized teacher framework to distill knowledge to a student network. In our approach, directed at face recognition use cases, we train four teachers on one specific ethnicity, leading to four highly specialized and biased teachers. Our strategy learns a project of these four teachers into a common space and distill that information to a student network. Our results highlighted increased performance and reduced bias for all our experiments. In addition, we further show that having biased/specialized teachers is crucial by showing that our approach achieves better results than when knowledge is distilled from four teachers trained on balanced datasets. Our approach represents a step forward to the understanding of the importance of ethnicity-specific features.