mlsanityData to 14 Oct 2026

Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessment

arXiv 2509.19659 ↗First posted 24 Sep 2025cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Large vision-language models (VLMs) can jointly interpret images and text, but they are also prone to absorbing and reproducing harmful social stereotypes when visual cues such as age, gender, race, clothing, or occupation are present. To investigate these risks, we introduce a news-image benchmark consisting of 1,343 image-question pairs drawn from diverse outlets, which we annotated with ground-truth answers and demographic attributes (age, gender, race, occupation, and sports). We evaluate a range of state-of-the-art VLMs and employ a large language model (LLM) as judge, with human verification. Our findings show that: (i) visual context systematically shifts model outputs in open-ended settings; (ii) bias prevalence varies across attributes and models, with particularly high risk for gender and occupation; and (iii) higher faithfulness does not necessarily correspond to lower bias. We release the benchmark prompts, evaluation rubric, and code to support reproducible and fairness-aware multimodal assessment.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 28 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

19 of 28 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing30 Jun
  2. 2024The Bias of Harmful Label Associations in Vision-Language Models11 Feb
  3. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images8 Feb
  4. 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples30 Nov
  5. 2023Survey of Social Bias in Vision-Language Models24 Sep
  6. 2023VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution21 Jun

Other references · 13

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  3. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  4. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  5. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  6. 2024Phi-4 Technical Reportcited by 711
  7. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  8. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  9. 2023Multimodal Large Language Models: A Surveycited by 195
  10. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  11. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 53
  12. 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Modelscited by 52
1 more
  1. 2024Enhancing Advanced Visual Reasoning Ability of Large Language Modelscited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025BioPro: Towards Difference-Aware Gender Fairness for Vision-Language Modelscited by 2
  2. 2026Cultural Counterfactuals: Evaluating Cultural Biases in Large Vision-Language Models with Counterfactual Examplescited by 2
  3. 2026LLM-as-Judge for Semantic Judging of Powerline Segmentation in UAV Inspectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals
  2. 2024Uncovering Bias in Large Vision-Language Models with Counterfactuals
  3. 2024A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models
  4. 2026FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making
  5. 2024VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model
  6. 2025Ask Me Again Differently: GRAS for Measuring Bias in Vision Language Models on Gender, Race, Age, and Skin Tone
  7. 2023MultiModal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision Language Models
  8. 2025VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models