mlsanityData to 7 Oct 2026

VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models

arXiv 2505.22897 ↗First posted 28 May 2025cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

While bias in large language models (LLMs) is well-studied, similar concerns in vision-language models (VLMs) have received comparatively less attention. Existing VLM bias studies often focus on portrait-style images and gender-occupation associations, overlooking broader and more complex social stereotypes and their implied harm. This work introduces VIGNETTE, a large-scale VQA benchmark with 30M+ images for evaluating bias in VLMs through a question-answering framework spanning four directions: factuality, perception, stereotyping, and decision making. Beyond narrowly-centered studies, we assess how VLMs interpret identities in contextualized settings, revealing how models make trait and capability assumptions and exhibit patterns of discrimination. Drawing from social psychology, we examine how VLMs connect visual identity cues to trait and role-based inferences, encoding social hierarchies, through biased selections. Our findings uncover subtle, multifaceted, and surprising stereotypical patterns, offering insights into how VLMs construct social meaning from inputs.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 34 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

18 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models10 Mar
  2. 2024Vision-Language Models Generate More Homogeneous Stories for Phenotypically Black Individuals12 Dec
  3. 2023Probing Intersectional Biases in Vision-Language Models with Counterfactual Examples4 Oct
  4. 2023DeAR: Debiasing Vision-Language Models with Additive Residuals18 Mar

Other references · 14

  1. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  2. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  3. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 426
  4. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  5. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
  6. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  7. 2020Measuring Social Biases in Grounded Vision and Language Embeddingscited by 61
  8. 2023Bias Against 93 Stigmatized Groups in Masked Language Models and Downstream Sentiment Classification Taskscited by 51
  9. 2022Gender and Racial Bias in Visual Question Answering Datasetscited by 39
  10. 2022American == White in Multimodal Language-and-Image AIcited by 39
  11. 2022Measuring Representational Harms in Image Captioningcited by 36
  12. 2024CounterCurate: Enhancing Physical and Semantic Visio-Linguistic Compositional Reasoning via Counterfactual Examplescited by 35
2 more
  1. 2024textttModSCAN: Measuring Stereotypical Bias in Large Vision-Language Models from Vision and Language Modalitiescited by 21
  2. 2024Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)cited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025DSO: Direct Steering Optimization for Bias Mitigationcited by 3
  2. 2026DeepBias: Adaptive In-depth Probing of Social Biases in LVLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024BiasDora: Exploring Hidden Biased Associations in Vision-Language Models
  2. 2024A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models
  3. 2026Edu-MMBias: A Three-Tier Multimodal Benchmark for Auditing Social Bias in Vision-Language Models under Educational Contexts
  4. 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals
  5. 2025Visual Cues of Gender and Race are Associated with Stereotyping in Vision-Language Models
  6. 2024More Distinctively Black and Feminine Faces Lead to Increased Stereotyping in Vision-Language Models
  7. 2025How Reasoning Influences Intersectional Biases in Vision Language Models
  8. 2024Uncovering Bias in Large Vision-Language Models with Counterfactuals