# UDIS: Unsupervised Discovery of Bias in Deep Visual Recognition Models

arXiv: https://arxiv.org/abs/2110.15499 · HTML: https://mlsanity.com/paper/2110.15499/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 50 references that are in the map.
- First posted on arXiv 29 Oct 2021.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [A Survey on Deep Transfer Learning](https://mlsanity.com/paper/1808.01974/) (2018)
- [On Fairness and Calibration](https://mlsanity.com/paper/1709.02012/) (2017)
- [AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias](https://mlsanity.com/paper/1810.01943/) (2018)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)
- [Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation](https://mlsanity.com/paper/1911.11834/) (2019)
- [Real Time Image Saliency for Black Box Classifiers](https://mlsanity.com/paper/1705.07857/) (2017)
- [Women also Snowboard: Overcoming Bias in Captioning Models](https://mlsanity.com/paper/1803.09797/) (2018)
- [Avoiding Discrimination through Causal Reasoning](https://mlsanity.com/paper/1706.02744/) (2017)
- [No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World](https://mlsanity.com/paper/1711.08536/) (2017)
- [Does Object Recognition Work for Everyone?](https://mlsanity.com/paper/1906.02659/) (2019)
- [Explaining Image Classifiers by Counterfactual Generation](https://mlsanity.com/paper/1807.08024/) (2018)
- [REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets](https://mlsanity.com/paper/2004.07999/) (2020)
- [Convolutional Networks with Dense Connectivity](https://mlsanity.com/paper/2001.02394/) (2020)
- [FairVis: Visual Analytics for Discovering Intersectional Bias in Machine Learning](https://mlsanity.com/paper/1904.05419/) (2019)
- [The Hessian Penalty: A Weak Prior for Unsupervised Disentanglement](https://mlsanity.com/paper/2008.10599/) (2020)
- [Don't Judge an Object by Its Context: Learning to Overcome Contextual Bias](https://mlsanity.com/paper/2001.03152/) (2020)
- [Predictive Inequity in Object Detection](https://mlsanity.com/paper/1902.11097/) (2019)
- [On Formalizing Fairness in Prediction with Machine Learning](https://mlsanity.com/paper/1710.03184/) (2017)

## Built on by (most cited first, 23 of 23)

- [Discovering and Mitigating Visual Biases through Keyword Explanation](https://mlsanity.com/paper/2301.11104/) (2023)
- [Discover and Mitigate Unknown Biases with Debiasing Alternate Networks](https://mlsanity.com/paper/2207.10077/) (2022)
- [FACTS: First Amplify Correlations and Then Slice to Discover Bias](https://mlsanity.com/paper/2309.17430/) (2023)
- [Mitigating Dataset Bias by Using Per-sample Gradient](https://mlsanity.com/paper/2205.15704/) (2022)
- [Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Models](https://mlsanity.com/paper/2304.13855/) (2023)
- [Angler: Helping Machine Translation Practitioners Prioritize Model Improvements](https://mlsanity.com/paper/2304.05967/) (2023)
- [VisCUIT: Visual Auditor for Bias in CNN Image Classifier](https://mlsanity.com/paper/2204.05899/) (2022)
- [A View From Somewhere: Human-Centric Face Representations](https://mlsanity.com/paper/2303.17176/) (2023)
- [AttributionScanner: A Visual Analytics System for Model Validation with Metadata-Free Slice Finding](https://mlsanity.com/paper/2401.06462/) (2024)
- [VISLIX: An XAI Framework for Validating Vision Models with Slice Discovery and Analysis](https://mlsanity.com/paper/2505.03132/) (2025)
- [Information Removal at the bottleneck in Deep Neural Networks](https://mlsanity.com/paper/2210.00891/) (2022)
- [Classifier-to-Bias: Toward Unsupervised Automatic Bias Detection for Visual Classifiers](https://mlsanity.com/paper/2504.20902/) (2025)
- [Attention IoU: Examining Biases in CelebA using Attention Maps](https://mlsanity.com/paper/2503.19846/) (2025)
- [ICON2: Reliably Benchmarking Predictive Inequity in Object Detection](https://mlsanity.com/paper/2306.04482/) (2023)
- [ViG-Bias: Visually Grounded Bias Discovery and Mitigation](https://mlsanity.com/paper/2407.01996/) (2024)
- [Subgroups Matter for Robust Bias Mitigation](https://mlsanity.com/paper/2505.21363/) (2025)
- [Discovering Intersectional Bias via Directional Alignment in Face Recognition Embeddings](https://mlsanity.com/paper/2510.15520/) (2025)
- [Unsupervised bias discovery in medical image segmentation](https://mlsanity.com/paper/2309.00451/) (2023)
- [Language-assisted Vision Model Debugger: A Sample-Free Approach to Finding and Fixing Bugs](https://mlsanity.com/paper/2312.05588/) (2023)
- [Common-Sense Bias Modeling for Classification Tasks](https://mlsanity.com/paper/2401.13213/) (2024)
- [Auditing a Dutch Public Sector Risk Profiling Algorithm Using an Unsupervised Bias Detection Tool](https://mlsanity.com/paper/2502.01713/) (2025)
- [Effective Guidance for Model Attention with Simple Yes-no Annotations](https://mlsanity.com/paper/2410.22312/) (2024)
- [Identifying Latent Concepts and Structures for Generalized Category Discovery](https://mlsanity.com/paper/2607.00620/) (2026)

## Abstract (arXiv)

Deep learning models have been shown to learn spurious correlations from data that sometimes lead to systematic failures for certain subpopulations. Prior work has typically diagnosed this by crowdsourcing annotations for various protected attributes and measuring performance, which is both expensive to acquire and difficult to scale. In this work, we propose UDIS, an unsupervised algorithm for surfacing and analyzing such failure modes. UDIS identifies subpopulations via hierarchical clustering of dataset embeddings and surfaces systematic failure modes by visualizing low performing clusters along with their gradient-weighted class-activation maps. We show the effectiveness of UDIS in identifying failure modes in models trained for image classification on the CelebA and MSCOCO datasets.