mlsanityData to 7 Oct 2026

Ask Me Again Differently: GRAS for Measuring Bias in Vision Language Models on Gender, Race, Age, and Skin Tone

arXiv 2508.18989 ↗First posted 26 Aug 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

As Vision Language Models (VLMs) become integral to real-world applications, understanding their demographic biases is critical. We introduce GRAS, a benchmark for uncovering demographic biases in VLMs across gender, race, age, and skin tone, offering the most diverse coverage to date. We further propose the GRAS Bias Score, an interpretable metric for quantifying bias. We benchmark five state-of-the-art VLMs and reveal concerning bias levels, with the least biased model attaining a GRAS Bias Score of only 2 out of 100. Our findings also reveal a methodological insight: evaluating bias in VLMs with visual question answering (VQA) requires considering multiple formulations of a question. Our code, data, and evaluation results are publicly available.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 38 references that are in the map. First posted on arXiv 26 Aug 2025.

Builds on

7 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images8 Feb

Other references · 6

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  3. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  4. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
  5. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  6. 2024Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)cited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Out of the box age estimation through facial imagery: A Comprehensive Benchmark of Vision-Language Models vs. out-of-the-box Traditional Architecturescited by 5
  2. 2026Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts
  2. 2024A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models
  3. 2023MultiModal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision Language Models
  4. 2024VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model
  5. 2025Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessment
  6. 2024debiaSAE: Benchmarking and Mitigating Vision-Language Model Bias
  7. 2024GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing
  8. 2025VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models