# A Survey on Bias in Visual Datasets

arXiv: https://arxiv.org/abs/2107.07919 · HTML: https://mlsanity.com/paper/2107.07919/

## Key facts

- Cited by 84 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 114 references that are in the map.
- First posted on arXiv 16 Jul 2021.

## Builds on (most cited first, 30 of 48)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [The Creation and Detection of Deepfakes: A Survey](https://mlsanity.com/paper/2004.11138/) (2020)
- [Large image datasets: A pyrrhic win for computer vision?](https://mlsanity.com/paper/2006.16923/) (2020)
- [An introduction to Topological Data Analysis: fundamental and practical aspects for data scientists](https://mlsanity.com/paper/1710.04019/) (2017)
- [Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy](https://mlsanity.com/paper/1912.07726/) (2019)
- [No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World](https://mlsanity.com/paper/1711.08536/) (2017)
- [Identifying and Correcting Label Bias in Machine Learning](https://mlsanity.com/paper/1901.04966/) (2019)
- [REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets](https://mlsanity.com/paper/2004.07999/) (2020)
- [Towards a Critical Race Methodology in Algorithmic Fairness](https://mlsanity.com/paper/1912.03593/) (2019)
- [Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases](https://mlsanity.com/paper/2010.15052/) (2020)
- [Discrimination in Online Ad Delivery](https://mlsanity.com/paper/1301.6822/) (2013)
- [Demographic Bias in Biometrics: A Survey on an Emerging Challenge](https://mlsanity.com/paper/2003.02488/) (2020)
- [Don't Judge an Object by Its Context: Learning to Overcome Contextual Bias](https://mlsanity.com/paper/2001.03152/) (2020)
- [A Deeper Look at Dataset Bias](https://mlsanity.com/paper/1505.01257/) (2015)
- [A Comprehensive Study on Face Recognition Biases Beyond Demographics](https://mlsanity.com/paper/2103.01592/) (2021)
- [Face Recognition: Too Bias, or Not Too Bias?](https://mlsanity.com/paper/2002.06483/) (2020)
- [Towards Measuring Fairness in AI: the Casual Conversations Dataset](https://mlsanity.com/paper/2104.02821/) (2021)
- [Discovering Causal Signals in Images](https://mlsanity.com/paper/1605.08179/) (2016)
- [Diversity in Faces](https://mlsanity.com/paper/1901.10436/) (2019)

## Built on by (most cited first, 30 of 84)

- [Towards Unbounded Machine Unlearning](https://mlsanity.com/paper/2302.09880/) (2023)
- [Towards Adversarial Evaluations for Inexact Machine Unlearning](https://mlsanity.com/paper/2201.06640/) (2022)
- [X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning](https://mlsanity.com/paper/2311.18799/) (2023)
- [SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples](https://mlsanity.com/paper/2312.00825/) (2023)
- [Representation Bias in Data: A Survey on Identification and Resolution Techniques](https://mlsanity.com/paper/2203.11852/) (2022)
- [Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparities](https://mlsanity.com/paper/2301.11100/) (2023)
- [Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time](https://mlsanity.com/paper/2407.01851/) (2024)
- [Would Deep Generative Models Amplify Bias in Future Models?](https://mlsanity.com/paper/2404.03242/) (2024)
- [FLAC: Fairness-Aware Representation Learning by Suppressing Attribute-Class Associations](https://mlsanity.com/paper/2304.14252/) (2023)
- [Multi-dimensional discrimination in Law and Machine Learning -- A comparative overview](https://mlsanity.com/paper/2302.05995/) (2023)
- [Towards Fair Face Verification: An In-depth Analysis of Demographic Biases](https://mlsanity.com/paper/2307.10011/) (2023)
- [Metrics for Dataset Demographic Bias: A Case Study on Facial Expression Recognition](https://mlsanity.com/paper/2303.15889/) (2023)
- [Large-scale Bilingual Language-Image Contrastive Learning](https://mlsanity.com/paper/2203.14463/) (2022)
- [T2Vs Meet VLMs: A Scalable Multimodal Dataset for Visual Harmfulness Recognition](https://mlsanity.com/paper/2409.19734/) (2024)
- [Efficient Bias Mitigation Without Privileged Information](https://mlsanity.com/paper/2409.17691/) (2024)
- [Trusted Media Challenge Dataset and User Study](https://mlsanity.com/paper/2201.04788/) (2022)
- [Diagnostics for Deep Neural Networks with Automated Copy/Paste Attacks](https://mlsanity.com/paper/2211.10024/) (2022)
- [Visual Data Diagnosis and Debiasing with Concept Graphs](https://mlsanity.com/paper/2409.18055/) (2024)
- [Intrinsic Bias is Predicted by Pretraining Data and Correlates with Downstream Performance in Vision-Language Encoders](https://mlsanity.com/paper/2502.07957/) (2025)
- [A Survey of Dataset Refinement for Problems in Computer Vision Datasets](https://mlsanity.com/paper/2210.11717/) (2022)
- [Casual Conversations v2: Designing a large consent-driven dataset to measure algorithmic bias and robustness](https://mlsanity.com/paper/2211.05809/) (2022)
- [DSAP: Analyzing Bias Through Demographic Comparison of Datasets](https://mlsanity.com/paper/2312.14626/) (2023)
- [Limitations of Data-Driven Spectral Reconstruction -- An Optics-Aware Analysis](https://mlsanity.com/paper/2401.03835/) (2024)
- [Can the accuracy bias by facial hairstyle be reduced through balancing the training data?](https://mlsanity.com/paper/2405.20062/) (2024)
- [Less can be more: representational vs. stereotypical gender bias in facial expression recognition](https://mlsanity.com/paper/2406.17405/) (2024)
- [BAdd: Bias Mitigation through Bias Addition](https://mlsanity.com/paper/2408.11439/) (2024)
- [MAVias: Mitigate any Visual Bias](https://mlsanity.com/paper/2412.06632/) (2024)
- [VIPHY: Probing "Visible" Physical Commonsense Knowledge](https://mlsanity.com/paper/2209.07000/) (2022)
- [An investigation of challenges encountered when specifying training data and runtime monitors for safety critical ML applications](https://mlsanity.com/paper/2301.13476/) (2023)
- [Visual Objectification in Films: Towards a New AI Task for Video Interpretation](https://mlsanity.com/paper/2401.13296/) (2024)

## Abstract (arXiv)

Computer Vision (CV) has achieved remarkable results, outperforming humans in several tasks. Nonetheless, it may result in significant discrimination if not handled properly as CV systems highly depend on the data they are fed with and can learn and amplify biases within such data. Thus, the problems of understanding and discovering biases are of utmost importance. Yet, there is no comprehensive survey on bias in visual datasets. Hence, this work aims to: i) describe the biases that might manifest in visual datasets; ii) review the literature on methods for bias discovery and quantification in visual datasets; iii) discuss existing attempts to collect bias-aware visual datasets. A key conclusion of our study is that the problem of bias discovery and quantification in visual datasets is still open, and there is room for improvement in terms of both methods and the range of biases that can be addressed. Moreover, there is no such thing as a bias-free dataset, so scientists and practitioners must become aware of the biases in their datasets and make them explicit. To this end, we propose a checklist to spot different types of bias during visual dataset collection.