mlsanityData to 14 Oct 2026

More Distinctively Black and Feminine Faces Lead to Increased Stereotyping in Vision-Language Models

arXiv 2407.06194 ↗First posted 22 May 2024cs.CV cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Vision Language Models (VLMs), exemplified by GPT-4V, adeptly integrate text and vision modalities. This integration enhances Large Language Models' ability to mimic human perception, allowing them to process image inputs. Despite VLMs' advanced capabilities, however, there is a concern that VLMs inherit biases of both modalities in ways that make biases more pervasive and difficult to mitigate. Our study explores how VLMs perpetuate homogeneity bias and trait associations with regards to race and gender. When prompted to write stories based on images of human faces, GPT-4V describes subordinate racial and gender groups with greater homogeneity than dominant groups and relies on distinct, yet generally positive, stereotypes. Importantly, VLM stereotyping is driven by visual cues rather than group membership alone such that faces that are rated as more prototypically Black and feminine are subject to greater stereotyping. These findings suggest that VLMs may associate subtle visual cues related to racial and gender groups with stereotypes in ways that could be challenging to mitigate. We explore the underlying reasons behind this behavior and discuss its implications and emphasize the importance of addressing these biases as VLMs come to mirror human perception.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 57 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

24 of 57 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  9. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  10. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  11. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
12 more
  1. 2022A Survey on In-context Learningcited by 721
  2. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  3. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  4. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  5. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  6. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 253
  7. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 207
  8. 2023Social Biases through the Text-to-Image Generation Lenscited by 127
  9. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  10. 2024Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humanscited by 31
  11. 2023Smiling Women Pitching Down: Auditing Representational and Presentational Gender Biases in Image Generative AIcited by 19
  12. 2016Quantitative Evaluation of Gender Bias in Astronomical Publications from Citation Countscited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Visual Cues of Gender and Race are Associated with Stereotyping in Vision-Language Models
  2. 2024Vision-Language Models Generate More Homogeneous Stories for Phenotypically Black Individuals
  3. 2025VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models
  4. 2025A Stereotype Content Analysis on Color-related Social Bias in Large Vision Language Models
  5. 2024A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models
  6. 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models
  7. 2024Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspective
  8. 2025VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models