# Distill and De-bias: Mitigating Bias in Face Verification using Knowledge Distillation

arXiv: https://arxiv.org/abs/2112.09786 · HTML: https://mlsanity.com/paper/2112.09786/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 40 of its 74 references that are in the map.
- First posted on arXiv 17 Dec 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [MagFace: A Universal Representation for Face Recognition and Quality Assessment](https://mlsanity.com/paper/2103.06627/) (2021)
- [Learning without Memorizing](https://mlsanity.com/paper/1811.08051/) (2018)
- [Mitigate Bias in Face Recognition using Skewness-Aware Reinforcement Learning](https://mlsanity.com/paper/1911.10692/) (2019)
- [Turning a Blind Eye: Explicit Removal of Biases and Variation from Deep Neural Network Embeddings](https://mlsanity.com/paper/1809.02169/) (2018)
- [Demographic Bias in Biometrics: A Survey on an Emerging Challenge](https://mlsanity.com/paper/2003.02488/) (2020)
- [Investigating Bias and Fairness in Facial Expression Recognition](https://mlsanity.com/paper/2007.10075/) (2020)
- [Face Recognition: Too Bias, or Not Too Bias?](https://mlsanity.com/paper/2002.06483/) (2020)
- [Mitigating Face Recognition Bias via Group Adaptive Classifier](https://mlsanity.com/paper/2006.07576/) (2020)
- [Fair Feature Distillation for Visual Recognition](https://mlsanity.com/paper/2106.04411/) (2021)
- [Accuracy comparison across face recognition algorithms: Where are we on measuring race bias?](https://mlsanity.com/paper/1912.07398/) (2019)
- [Understanding and Mitigating Annotation Bias in Facial Expression Recognition](https://mlsanity.com/paper/2108.08504/) (2021)
- [Analysis of Gender Inequality In Face Recognition Accuracy](https://mlsanity.com/paper/2002.00065/) (2020)
- [An All-In-One Convolutional Neural Network for Face Analysis](https://mlsanity.com/paper/1611.00851/) (2016)
- [UMDFaces: An Annotated Face Dataset for Training Deep Networks](https://mlsanity.com/paper/1611.01484/) (2016)
- [Characterizing the Variability in Face Recognition Accuracy Relative to Race](https://mlsanity.com/paper/1904.07325/) (2019)
- [Consistent Instance False Positive Improves Fairness in Face Recognition](https://mlsanity.com/paper/2106.05519/) (2021)
- [PASS: Protected Attribute Suppression System for Mitigating Bias in Face Recognition](https://mlsanity.com/paper/2108.03764/) (2021)
- [How Does Gender Balance In Training Data Affect Face Recognition Accuracy?](https://mlsanity.com/paper/2002.02934/) (2020)
- [An Experimental Evaluation of Covariates Effects on Unconstrained Face Verification](https://mlsanity.com/paper/1808.05508/) (2018)
- [Deep Learning for Face Recognition: Pride or Prejudiced?](https://mlsanity.com/paper/1904.01219/) (2019)
- [Is Face Recognition Sexist? No, Gendered Hairstyles and Biology Are](https://mlsanity.com/paper/2008.06989/) (2020)

## Built on by (most cited first, 6 of 6)

- [Racial Bias within Face Recognition: A Survey](https://mlsanity.com/paper/2305.00817/) (2023)
- [Gradient Attention Balance Network: Mitigating Face Recognition Racial Bias via Gradient Attention](https://mlsanity.com/paper/2304.02284/) (2023)
- [Gaussian Harmony: Attaining Fairness in Diffusion-based Face Generation Models](https://mlsanity.com/paper/2312.14976/) (2023)
- [Dual-Teacher De-biasing Distillation Framework for Multi-domain Fake News Detection](https://mlsanity.com/paper/2312.01006/) (2023)
- [Biasing & Debiasing based Approach Towards Fair Knowledge Transfer for Equitable Skin Analysis](https://mlsanity.com/paper/2405.10256/) (2024)
- [On the Illusion of Gender Bias in Face Recognition: Explaining the Fairness Issue Through Non-demographic Attributes](https://mlsanity.com/paper/2501.12020/) (2025)

## Abstract (arXiv)

Face recognition networks generally demonstrate bias with respect to sensitive attributes like gender, skintone etc. For gender and skintone, we observe that the regions of the face that a network attends to vary by the category of an attribute. This might contribute to bias. Building on this intuition, we propose a novel distillation-based approach called Distill and De-bias (D&D) to enforce a network to attend to similar face regions, irrespective of the attribute category. In D&D, we train a teacher network on images from one category of an attribute; e.g. light skintone. Then distilling information from the teacher, we train a student network on images of the remaining category; e.g., dark skintone. A feature-level distillation loss constrains the student network to generate teacher-like representations. This allows the student network to attend to similar face regions for all attribute categories and enables it to reduce bias. We also propose a second distillation step on top of D&D, called D&D++. Here, we distill the `un-biasedness' of the D&D network into a new student network, the D&D++ network, while training this new network on all attribute categories; e.g., both light and dark skintones. This helps us train a network that is less biased for an attribute, while obtaining higher face verification performance than D&D. We show that D&D++ outperforms existing baselines in reducing gender and skintone bias on the IJB-C dataset, while obtaining higher face verification performance than existing adversarial de-biasing methods. We evaluate the effectiveness of our proposed methods on two state-of-the-art face recognition networks: ArcFace and Crystalface.