# Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals

arXiv: https://arxiv.org/abs/2405.20152 · HTML: https://mlsanity.com/paper/2405.20152/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 30 May 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Social bias and fairness in VLMs](https://mlsanity.com/mllm/social-bias-and-fairness-in-vlms/)

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- ["I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://mlsanity.com/paper/2205.09209/) (2022)
- [Counterfactual Fairness in Text Classification through Robustness](https://mlsanity.com/paper/1809.10610/) (2018)
- [Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search](https://mlsanity.com/paper/1905.01989/) (2019)
- [How well can Text-to-Image Generative Models understand Ethical Natural Language Interventions?](https://mlsanity.com/paper/2210.15230/) (2022)
- [Toward Gender-Inclusive Coreference Resolution](https://mlsanity.com/paper/1910.13913/) (2019)
- [Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics](https://mlsanity.com/paper/2106.14574/) (2021)
- [DeAR: Debiasing Vision-Language Models with Additive Residuals](https://mlsanity.com/paper/2303.10431/) (2023)
- [Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images](https://mlsanity.com/paper/2402.05779/) (2024)
- [SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples](https://mlsanity.com/paper/2312.00825/) (2023)
- [The Use and Misuse of Counterfactuals in Ethical Machine Learning](https://mlsanity.com/paper/2102.05085/) (2021)
- [Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2104.08666/) (2021)
- [VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution](https://mlsanity.com/paper/2306.12424/) (2023)
- [Bias Against 93 Stigmatized Groups in Masked Language Models and Downstream Sentiment Classification Tasks](https://mlsanity.com/paper/2306.05550/) (2023)
- [On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Research](https://mlsanity.com/paper/2304.12397/) (2023)
- [MultiModal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision Language Models](https://mlsanity.com/paper/2303.12734/) (2023)
- [Computer Vision and Conflicting Values: Describing People with Automated Alt Text](https://mlsanity.com/paper/2105.12754/) (2021)
- [LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Model](https://mlsanity.com/paper/2404.01331/) (2024)

## Built on by (most cited first, 28 of 28)

- [Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers](https://mlsanity.com/paper/2506.23918/) (2025)
- [GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing](https://mlsanity.com/paper/2407.00600/) (2024)
- [Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)](https://mlsanity.com/paper/2410.19314/) (2024)
- [BiasDora: Exploring Hidden Biased Associations in Vision-Language Models](https://mlsanity.com/paper/2407.02066/) (2024)
- [Fairness Definitions in Language Models Explained](https://mlsanity.com/paper/2407.18454/) (2024)
- [FairI Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypes](https://mlsanity.com/paper/2506.23111/) (2025)
- [Debiasing Large Vision-Language Models by Ablating Protected Attribute Representations](https://mlsanity.com/paper/2410.13976/) (2024)
- [Interpretable Debiasing of Vision-Language Models for Social Fairness](https://mlsanity.com/paper/2602.24014/) (2026)
- [Lookism: The overlooked bias in computer vision](https://mlsanity.com/paper/2408.11448/) (2024)
- [LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compression](https://mlsanity.com/paper/2503.04982/) (2025)
- [Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Models](https://mlsanity.com/paper/2510.03721/) (2025)
- [When Tom Eats Kimchi: Evaluating Cultural Bias of Multimodal Large Language Models in Cultural Mixture Contexts](https://mlsanity.com/paper/2503.16826/) (2025)
- [When Algorithms Play Favorites: Lookism in the Generation and Perception of Faces](https://mlsanity.com/paper/2506.11025/) (2025)
- [Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos](https://mlsanity.com/paper/2601.06931/) (2026)
- [Aesthetics as Structural Harm: Algorithmic Lookism Across Text-to-Image Generation and Classification](https://mlsanity.com/paper/2601.11651/) (2026)
- [Cultural Counterfactuals: Evaluating Cultural Biases in Large Vision-Language Models with Counterfactual Examples](https://mlsanity.com/paper/2603.02370/) (2026)
- [A Stereotype Content Analysis on Color-related Social Bias in Large Vision Language Models](https://mlsanity.com/paper/2505.20901/) (2025)
- [Interpreting Social Bias in LVLMs via Information Flow Analysis and Multi-Round Dialogue Evaluation](https://mlsanity.com/paper/2505.21106/) (2025)
- [Happy Young Women, Grumpy Old Men? Emotion-Driven Demographic Biases in Synthetic Face Generation](https://mlsanity.com/paper/2602.00032/) (2026)
- [Debias your Large Multi-Modal Model at Test-Time via Non-Contrastive Visual Attribute Steering](https://mlsanity.com/paper/2411.12590/) (2024)
- [Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictions](https://mlsanity.com/paper/2603.28387/) (2026)
- [Cross-Cultural Value Attribution in Large Vision-Language Models](https://mlsanity.com/paper/2604.09945/) (2026)
- [Symphony of Bias: Exploring Gender Associations with Musical Instruments in Multimodal LLMs](https://mlsanity.com/paper/2607.26355/) (2026)
- [SportsGrounder: Proposal-Aided Interleaved Grounding for Dense Sports Video Reasoning](https://mlsanity.com/paper/2608.07932/) (2026)
- [ViD: Vision-Dominant Gender Bias Mitigation for Large Vision-Language Models](https://mlsanity.com/paper/2609.16647/) (2026)
- [Visual Input and Its Framing Affect Attribute-based Descriptions Produced by Large Vision-Language Models](https://mlsanity.com/paper/2609.18345/) (2026)
- [Same Chart, Different Story: Bias in Vision-Language Chart Interpretation](https://mlsanity.com/paper/2609.26210/) (2026)
- [Gender Bias in Vision-Language In-Context Learning](https://mlsanity.com/paper/2609.27682/) (2026)

## Abstract (arXiv)

With the advent of Large Language Models (LLMs) possessing increasingly impressive capabilities, a number of Large Vision-Language Models (LVLMs) have been proposed to augment LLMs with visual inputs. Such models condition generated text on both an input image and a text prompt, enabling a variety of use cases such as visual question answering and multimodal chat. While prior studies have examined the social biases contained in text generated by LLMs, this topic has been relatively unexplored in LVLMs. Examining social biases in LVLMs is particularly challenging due to the confounding contributions of bias induced by information contained across the text and visual modalities. To address this challenging problem, we conduct a large-scale study of text generated by different LVLMs under counterfactual changes to input images, producing over 57 million responses from popular models. Our multi-dimensional bias evaluation framework reveals that social attributes such as perceived race, gender, and physical characteristics depicted in images can significantly influence the generation of toxic content, competency-associated words, harmful stereotypes, and numerical ratings of individuals.