# Balancing the Picture: Debiasing Vision-Language Datasets with Synthetic Contrast Sets

arXiv: https://arxiv.org/abs/2305.15407 · HTML: https://mlsanity.com/paper/2305.15407/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 49 of its 66 references that are in the map.
- First posted on arXiv 24 May 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Social bias and fairness in VLMs](https://mlsanity.com/mllm/social-bias-and-fairness-in-vlms/)

## Builds on (most cited first, 30 of 49)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [Women also Snowboard: Overcoming Bias in Captioning Models](https://mlsanity.com/paper/1803.09797/) (2018)
- [Stable Bias: Analyzing Societal Representations in Diffusion Models](https://mlsanity.com/paper/2303.11408/) (2023)
- [Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy](https://mlsanity.com/paper/1912.07726/) (2019)
- [Does Object Recognition Work for Everyone?](https://mlsanity.com/paper/1906.02659/) (2019)
- [Fair Diffusion: Instructing Text-to-Image Generation Models on Fairness](https://mlsanity.com/paper/2302.10893/) (2023)
- [Debiasing Vision-Language Models via Biased Prompts](https://mlsanity.com/paper/2302.00070/) (2023)
- [Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search](https://mlsanity.com/paper/1905.01989/) (2019)
- [Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications](https://mlsanity.com/paper/2108.02818/) (2021)

## Built on by (most cited first, 27 of 27)

- [An Introduction to Vision-Language Modeling](https://mlsanity.com/paper/2405.17247/) (2024)
- [Large-scale Reinforcement Learning for Diffusion Models](https://mlsanity.com/paper/2401.12244/) (2024)
- [SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples](https://mlsanity.com/paper/2312.00825/) (2023)
- [VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution](https://mlsanity.com/paper/2306.12424/) (2023)
- [No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models](https://mlsanity.com/paper/2405.13777/) (2024)
- [PrefPalette: Personalized Preference Modeling with Latent Attributes](https://mlsanity.com/paper/2507.13541/) (2025)
- [A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models](https://mlsanity.com/paper/2307.12980/) (2023)
- [AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessment](https://mlsanity.com/paper/2509.11620/) (2025)
- [Social Perception of Faces in a Vision-Language Model](https://mlsanity.com/paper/2408.14435/) (2024)
- [Diverse Diffusion: Enhancing Image Diversity in Text-to-Image Generation](https://mlsanity.com/paper/2310.12583/) (2023)
- [From Head to Tail: Towards Balanced Representation in Large Vision-Language Models through Adaptive Data Calibration](https://mlsanity.com/paper/2503.12821/) (2025)
- [Resampled Datasets Are Not Enough: Mitigating Societal Bias Beyond Single Attributes](https://mlsanity.com/paper/2407.03623/) (2024)
- [Debiasing Large Vision-Language Models by Ablating Protected Attribute Representations](https://mlsanity.com/paper/2410.13976/) (2024)
- [Vision-Language Models display a strong gender bias](https://mlsanity.com/paper/2508.11262/) (2025)
- [Limitations of Face Image Generation](https://mlsanity.com/paper/2309.07277/) (2023)
- [Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Models](https://mlsanity.com/paper/2510.03721/) (2025)
- [FairCoT: Enhancing Fairness in Text-to-Image Generation via Chain of Thought Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2406.09070/) (2024)
- [They're All Doctors: Synthesizing Diverse Counterfactuals to Mitigate Associative Bias](https://mlsanity.com/paper/2406.11331/) (2024)
- [Evaluating Model Bias Requires Characterizing its Mistakes](https://mlsanity.com/paper/2407.10633/) (2024)
- [A Picture is Worth 500 Labels: A Case Study of Demographic Disparities in Local Machine Learning Models for Instagram and TikTok](https://mlsanity.com/paper/2403.19717/) (2024)
- [A Taxonomy of Challenges to Curating Fair Datasets](https://mlsanity.com/paper/2406.06407/) (2024)
- [Is What You Ask For What You Get? Investigating Concept Associations in Text-to-Image Models](https://mlsanity.com/paper/2410.04634/) (2024)
- [Do Image-Text Metrics Respect Semantic Invariances?](https://mlsanity.com/paper/2605.24702/) (2026)
- [Offline Evaluation of Set-Based Text-to-Image Generation](https://mlsanity.com/paper/2410.17331/) (2024)
- [How Bias Binds: Measuring Hidden Associations for Bias Control in Text-to-Image Compositions](https://mlsanity.com/paper/2511.07091/) (2025)
- [Debias your Large Multi-Modal Model at Test-Time via Non-Contrastive Visual Attribute Steering](https://mlsanity.com/paper/2411.12590/) (2024)
- [Anchoring Bias: A Persistent Fairness Backdoor Attack against MLLMs under Continual Learning](https://mlsanity.com/paper/2608.21577/) (2026)

## Abstract (arXiv)

Vision-language models are growing in popularity and public visibility to generate, edit, and caption images at scale; but their outputs can perpetuate and amplify societal biases learned during pre-training on uncurated image-text pairs from the internet. Although debiasing methods have been proposed, we argue that these measurements of model bias lack validity due to dataset bias. We demonstrate there are spurious correlations in COCO Captions, the most commonly used dataset for evaluating bias, between background context and the gender of people in-situ. This is problematic because commonly-used bias metrics (such as Bias@K) rely on per-gender base rates. To address this issue, we propose a novel dataset debiasing pipeline to augment the COCO dataset with synthetic, gender-balanced contrast sets, where only the gender of the subject is edited and the background is fixed. However, existing image editing methods have limitations and sometimes produce low-quality images; so, we introduce a method to automatically filter the generated images based on their similarity to real images. Using our balanced synthetic contrast sets, we benchmark bias in multiple CLIP-based models, demonstrating how metrics are skewed by imbalance in the original COCO images. Our results indicate that the proposed approach improves the validity of the evaluation, ultimately contributing to more realistic understanding of bias in vision-language models.