# Gender Slopes: Counterfactual Fairness for Computer Vision Models by Attribute Manipulation

arXiv: https://arxiv.org/abs/2005.10430 · HTML: https://mlsanity.com/paper/2005.10430/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2020.

## Built on by (most cited first, 30 of 37)

- [Understanding and Mitigating Annotation Bias in Facial Expression Recognition](https://mlsanity.com/paper/2108.08504/) (2021)
- [Discover and Mitigate Unknown Biases with Debiasing Alternate Networks](https://mlsanity.com/paper/2207.10077/) (2022)
- [Discover the Unknown Biased Attribute of an Image Classifier](https://mlsanity.com/paper/2104.14556/) (2021)
- [Finding and Fixing Spurious Patterns with Explanations](https://mlsanity.com/paper/2106.02112/) (2021)
- [FairGRAPE: Fairness-aware GRAdient Pruning mEthod for Face Attribute Classification](https://mlsanity.com/paper/2207.10888/) (2022)
- [Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition](https://mlsanity.com/paper/2210.09943/) (2022)
- [Survey of Social Bias in Vision-Language Models](https://mlsanity.com/paper/2309.14381/) (2023)
- [UDIS: Unsupervised Discovery of Bias in Deep Visual Recognition Models](https://mlsanity.com/paper/2110.15499/) (2021)
- [Latent Space Smoothing for Individually Fair Representations](https://mlsanity.com/paper/2111.13650/) (2021)
- [Anatomizing Bias in Facial Analysis](https://mlsanity.com/paper/2112.06522/) (2021)
- [Explaining Deep Convolutional Neural Networks via Latent Visual-Semantic Filter Attention](https://mlsanity.com/paper/2204.04601/) (2022)
- [Counterfactually Fair Representation](https://mlsanity.com/paper/2311.05420/) (2023)
- [Distributionally Generative Augmentation for Fair Facial Attribute Classification](https://mlsanity.com/paper/2403.06606/) (2024)
- [Resampled Datasets Are Not Enough: Mitigating Societal Bias Beyond Single Attributes](https://mlsanity.com/paper/2407.03623/) (2024)
- [What's in a Caption? Dataset-Specific Linguistic Diversity and Its Effect on Visual Description Models and Metrics](https://mlsanity.com/paper/2205.06253/) (2022)
- [Distribution Aware Metrics for Conditional Natural Language Generation](https://mlsanity.com/paper/2209.07518/) (2022)
- [Finding Differences Between Transformers and ConvNets Using Counterfactual Simulation Testing](https://mlsanity.com/paper/2211.16499/) (2022)
- [AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Data](https://mlsanity.com/paper/2503.05665/) (2025)
- [Manifestations of Xenophobia in AI Systems](https://mlsanity.com/paper/2212.07877/) (2022)
- [Enhancing Fairness of Visual Attribute Predictors](https://mlsanity.com/paper/2207.05727/) (2022)
- [Counterfactual Generation Under Confounding](https://mlsanity.com/paper/2210.12368/) (2022)
- [FaceX: Understanding Face Attribute Classifiers through Summary Model Explanations](https://mlsanity.com/paper/2412.07313/) (2024)
- [Do Fairness Interventions Come at the Cost of Privacy: Evaluations for Binary Classifiers](https://mlsanity.com/paper/2503.06150/) (2025)
- [AI Alignment in Medical Imaging: Unveiling Hidden Biases Through Counterfactual Analysis](https://mlsanity.com/paper/2504.19621/) (2025)
- [Discovering Intersectional Bias via Directional Alignment in Face Recognition Embeddings](https://mlsanity.com/paper/2510.15520/) (2025)
- [Adaptation and Generalization for Unknown Sensitive Factors of Variations](https://mlsanity.com/paper/2107.13625/) (2021)
- [On Counterfactual Data Augmentation Under Confounding](https://mlsanity.com/paper/2305.18183/) (2023)
- [Approximating Discrimination Within Models When Faced With Several Non-Binary Sensitive Attributes](https://mlsanity.com/paper/2408.06099/) (2024)
- [Constructing Fair Latent Space for Intersection of Fairness and Explainability](https://mlsanity.com/paper/2412.17523/) (2024)
- [Causality in Neural Networks -- An Extended Abstract](https://mlsanity.com/paper/2106.05842/) (2021)

## Abstract (arXiv)

Automated computer vision systems have been applied in many domains including security, law enforcement, and personal devices, but recent reports suggest that these systems may produce biased results, discriminating against people in certain demographic groups. Diagnosing and understanding the underlying true causes of model biases, however, are challenging tasks because modern computer vision systems rely on complex black-box models whose behaviors are hard to decode. We propose to use an encoder-decoder network developed for image attribute manipulation to synthesize facial images varying in the dimensions of gender and race while keeping other signals intact. We use these synthesized images to measure counterfactual fairness of commercial computer vision classifiers by examining the degree to which these classifiers are affected by gender and racial cues controlled in the images, e.g., feminine faces may elicit higher scores for the concept of nurse and lower scores for STEM-related concepts. We also report the skewed gender representations in an online search service on profession-related keywords, which may explain the origin of the biases encoded in the models.