# Fair Feature Distillation for Visual Recognition

arXiv: https://arxiv.org/abs/2106.04411 · HTML: https://mlsanity.com/paper/2106.04411/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 45 references that are in the map.
- First posted on arXiv 27 May 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [A Reductions Approach to Fair Classification](https://mlsanity.com/paper/1803.02453/) (2018)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017)
- [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019)
- [AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias](https://mlsanity.com/paper/1810.01943/) (2018)
- [Fairness in Criminal Justice Risk Assessments: The State of the Art](https://mlsanity.com/paper/1703.09207/) (2017)
- [Deep Face Recognition: A Survey](https://mlsanity.com/paper/1804.06655/) (2018)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [The Variational Fair Autoencoder](https://mlsanity.com/paper/1511.00830/) (2015)
- [Learning Deep Representations with Probabilistic Knowledge Transfer](https://mlsanity.com/paper/1803.10837/) (2018)
- [Flexibly Fair Representation Learning by Disentanglement](https://mlsanity.com/paper/1906.02589/) (2019)
- [Identifying and Correcting Label Bias in Machine Learning](https://mlsanity.com/paper/1901.04966/) (2019)
- [Mitigate Bias in Face Recognition using Skewness-Aware Reinforcement Learning](https://mlsanity.com/paper/1911.10692/) (2019)

## Built on by (most cited first, 30 of 59)

- [Fair Contrastive Learning for Facial Attribute Classification](https://mlsanity.com/paper/2203.16209/) (2022)
- [Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifiers](https://mlsanity.com/paper/2301.11578/) (2023)
- [Learning Fair Classifiers with Partially Annotated Group Labels](https://mlsanity.com/paper/2111.14581/) (2021)
- [On Learning Fairness and Accuracy on Multiple Subgroups](https://mlsanity.com/paper/2210.10837/) (2022)
- [Conditional Contrastive Learning for Improving Fairness in Self-Supervised Learning](https://mlsanity.com/paper/2106.02866/) (2021)
- [FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSs](https://mlsanity.com/paper/2403.15593/) (2024)
- [Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Survey](https://mlsanity.com/paper/2211.05617/) (2022)
- [Re-weighting Based Group Fairness Regularization via Classwise Robust Optimization](https://mlsanity.com/paper/2303.00442/) (2023)
- [Fast and Efficient MMD-based Fair PCA via Optimization over Stiefel Manifold](https://mlsanity.com/paper/2109.11196/) (2021)
- [Anatomizing Bias in Facial Analysis](https://mlsanity.com/paper/2112.06522/) (2021)
- [Resilient Constrained Learning](https://mlsanity.com/paper/2306.02426/) (2023)
- [Toward Fairness Through Fair Multi-Exit Framework for Dermatological Disease Diagnosis](https://mlsanity.com/paper/2306.14518/) (2023)
- [Fairness and Bias Mitigation in Computer Vision: A Survey](https://mlsanity.com/paper/2408.02464/) (2024)
- [Semantic Scene Completion with Cleaner Self](https://mlsanity.com/paper/2303.09977/) (2023)
- [Achieve Fairness without Demographics for Dermatological Disease Diagnosis](https://mlsanity.com/paper/2401.08066/) (2024)
- [Racial Bias within Face Recognition: A Survey](https://mlsanity.com/paper/2305.00817/) (2023)
- [Achieving Fairness Through Channel Pruning for Dermatological Disease Diagnosis](https://mlsanity.com/paper/2405.08681/) (2024)
- [CoupleFace: Relation Matters for Face Recognition Distillation](https://mlsanity.com/paper/2204.05502/) (2022)
- [A Multidimensional Analysis of Social Biases in Vision Transformers](https://mlsanity.com/paper/2308.01948/) (2023)
- [Distill and De-bias: Mitigating Bias in Face Verification using Knowledge Distillation](https://mlsanity.com/paper/2112.09786/) (2021)
- [Fairness-aware Vision Transformer via Debiased Self-Attention](https://mlsanity.com/paper/2301.13803/) (2023)
- [A Large-Scale Empirical Study on Improving the Fairness of Image Classification Models](https://mlsanity.com/paper/2401.03695/) (2024)
- [Incorporating Rather Than Eliminating: Achieving Fairness for Skin Disease Diagnosis Through Group-Specific Expert](https://mlsanity.com/paper/2506.17787/) (2025)
- [Fair Multi-Exit Framework for Facial Attribute Classification](https://mlsanity.com/paper/2301.02989/) (2023)
- [What is Left After Distillation? How Knowledge Transfer Impacts Fairness and Bias](https://mlsanity.com/paper/2410.08407/) (2024)
- [Enhancing Fairness of Visual Attribute Predictors](https://mlsanity.com/paper/2207.05727/) (2022)
- [Backdoor for Debias: Mitigating Model Bias with Backdoor Attack-based Artificial Bias](https://mlsanity.com/paper/2303.01504/) (2023)
- [Boosting Residual Networks with Group Knowledge](https://mlsanity.com/paper/2308.13772/) (2023)
- [LumiNet: Perception-Driven Knowledge Distillation via Statistical Logit Calibration](https://mlsanity.com/paper/2310.03669/) (2023)
- [FairDD: Fair Dataset Distillation](https://mlsanity.com/paper/2411.19623/) (2024)

## Abstract (arXiv)

Fairness is becoming an increasingly crucial issue for computer vision, especially in the human-related decision systems. However, achieving algorithmic fairness, which makes a model produce indiscriminative outcomes against protected groups, is still an unresolved problem. In this paper, we devise a systematic approach which reduces algorithmic biases via feature distillation for visual recognition tasks, dubbed as MMD-based Fair Distillation (MFD). While the distillation technique has been widely used in general to improve the prediction accuracy, to the best of our knowledge, there has been no explicit work that also tries to improve fairness via distillation. Furthermore, We give a theoretical justification of our MFD on the effect of knowledge distillation and fairness. Throughout the extensive experiments, we show our MFD significantly mitigates the bias against specific minorities without any loss of the accuracy on both synthetic and real-world face datasets.