mlsanityData to 14 Oct 2026

Vision-Language Models display a strong gender bias

arXiv 2508.11262 ↗First posted 15 Aug 2025cs.CV cs.AIcited by 5

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Vision-language models (VLM) align images and text in a shared representation space that is useful for retrieval and zero-shot transfer. Yet, this alignment can encode and amplify social stereotypes in subtle ways that are not obvious from standard accuracy metrics. In this study, we test whether the contrastive vision-language encoder exhibits gender-linked associations when it places embeddings of face images near embeddings of short phrases that describe occupations and activities. We assemble a dataset of 220 face photographs split by perceived binary gender and a set of 150 unique statements distributed across six categories covering emotional labor, cognitive labor, domestic labor, technical labor, professional roles, and physical labor. We compute unit-norm image embeddings for every face and unit-norm text embeddings for every statement, then define a statement-level association score as the difference between the mean cosine similarity to the male set and the mean cosine similarity to the female set, where positive values indicate stronger association with the male set and negative values indicate stronger association with the female set. We attach bootstrap confidence intervals by resampling images within each gender group, aggregate by category with a separate bootstrap over statements, and run a label-swap null model that estimates the level of mean absolute association we would expect if no gender structure were present. The outcome is a statement-wise and category-wise map of gender associations in a contrastive vision-language space, accompanied by uncertainty, simple sanity checks, and a robust gender bias evaluation framework.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 30 references that are in the map. First posted on arXiv 15 Aug 2025.

Builds on

21 of 30 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Identifying Implicit Social Biases in Vision-Language Models1 Nov
  2. 2024GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing30 Jun
  3. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images8 Feb
  4. 2023Survey of Social Bias in Vision-Language Models24 Sep
  5. 2023Balancing the Picture: Debiasing Vision-Language Datasets with Synthetic Contrast Sets24 May
  6. 2023Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparities26 Jan

Other references · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
  3. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  4. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  5. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  6. 2018Gender Bias in Neural Natural Language Processingcited by 211
  7. 2023Gender bias and stereotypes in Large Language Modelscited by 187
  8. 2020REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasetscited by 136
  9. 2022Unsupervised Prompt Learning for Vision-Language Modelscited by 127
  10. 2024Unveiling Encoder-Free Vision-Language Modelscited by 94
  11. 2019FairFace: Face Attribute Dataset for Balanced Race, Gender, and Agecited by 87
  12. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 68
3 more
  1. 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Modelscited by 30
  2. 2024Can We Talk Models Into Seeing the World Differently?cited by 23
  3. 2023Evaluating Biased Attitude Associations of Language Models in an Intersectional Contextcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Surgeons Are Indian Males and Speech Therapists Are White Females: Auditing Biases in Vision-Language Models for Healthcare Professionalscited by 1
  2. 2026Vision-Language Models Suppress Female Representations Under Ambiguous Inputcited by 1
  3. 2026Fairness-Aware Test-Time Prompt Tuningcited by 1
  4. 2026Auditing Disability Representation in Vision-Language Modelscited by 0
  5. 2026Knowing When Not to Answer: Abstention and Refusal Reasoning in Vision--Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Data Matters Most: Auditing Social Bias in Contrastive Vision Language Models
  2. 2025Breaking Language Barriers or Reinforcing Bias? A Study of Gender and Racial Disparities in Multilingual Contrastive Vision Language Models
  3. 2023VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution
  4. 2024Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspective
  5. 2025Visual Cues of Gender and Race are Associated with Stereotyping in Vision-Language Models
  6. 2024Social Perception of Faces in a Vision-Language Model
  7. 2024A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models
  8. 2023Probing Intersectional Biases in Vision-Language Models with Counterfactual Examples