mlsanityData to 7 Oct 2026

VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model

arXiv 2406.14194 ↗First posted 20 Jun 2024cs.CV cs.AIcited by 26

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

The emergence of Large Vision-Language Models (LVLMs) marks significant strides towards achieving general artificial intelligence. However, these advancements are accompanied by concerns about biased outputs, a challenge that has yet to be thoroughly explored. Existing benchmarks are not sufficiently comprehensive in evaluating biases due to their limited data scale, single questioning format and narrow sources of bias. To address this problem, we introduce VLBiasBench, a comprehensive benchmark designed to evaluate biases in LVLMs. VLBiasBench, features a dataset that covers nine distinct categories of social biases, including age, disability status, gender, nationality, physical appearance, race, religion, profession, social economic status, as well as two intersectional bias categories: race x gender and race x social economic status. To build a large-scale dataset, we use Stable Diffusion XL model to generate 46,848 high-quality images, which are combined with various questions to creat 128,342 samples. These questions are divided into open-ended and close-ended types, ensuring thorough consideration of bias sources and a comprehensive evaluation of LVLM biases from multiple perspectives. We conduct extensive evaluations on 15 open-source models as well as two advanced closed-source models, yielding new insights into the biases present in these models. Our benchmark is available at https://github.com/Xiangkui-Cao/VLBiasBench.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 78 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

26 of 78 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Configurable Fairness: Direct Optimization of Parity Metrics via Vision-Language Models15 Mar
  2. 2024A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models21 Feb
  3. 2023MultiModal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision Language Models16 Mar

Other references · 23

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  7. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  8. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  10. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  11. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  12. 2024Yi: Open Foundation Models by 01.AIcited by 706
11 more
  1. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  2. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  3. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  4. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  5. 2022Red-Teaming the Stable Diffusion Safety Filtercited by 257
  6. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  7. 2019Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddingscited by 146
  8. 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
  9. 2024FairCLIP: Harnessing Fairness in Vision-Language Learningcited by 69
  10. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  11. 2024B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructionscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 11; 2026: 10.

26 papers.

Sort:

By area:

  1. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  2. 2024GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probingcited by 31
  3. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluationcited by 19
  4. 2024Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)cited by 15
  5. 2025Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimizationcited by 14
  6. 2025A Large Scale Analysis of Gender Biases in Text-to-Image Generative Modelscited by 12
  7. 2025BBQ-V: Benchmarking Visual Stereotype Bias in Large Multimodal Modelscited by 10
  8. 2024Social Debiasing for Fair Multi-modal LLMscited by 8
  9. 2025VLDBench Evaluating Multimodal Disinformation with Regulatory Alignmentcited by 8
  10. 2025No Free Lunch in Language Model Bias Mitigation? Targeted Bias Reduction Can Exacerbate Unmitigated LLM Biasescited by 6
  11. 2025Bias in Gender Bias Benchmarks: How Spurious Features Distort Evaluationcited by 5
  12. 2025FairReason: Balancing Reasoning and Social Bias in MLLMscited by 2
14 more
  1. 2025Bias Beyond Demographics: Probing Decision Boundaries in Black-Box LVLMs via Counterfactual VQAcited by 2
  2. 2025Beyond the Vision Encoder: Identifying and Mitigating Spatial Bias in Large Vision-Language Modelscited by 2
  3. 2026Cultural Counterfactuals: Evaluating Cultural Biases in Large Vision-Language Models with Counterfactual Examplescited by 2
  4. 2026Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMscited by 1
  5. 2024Debias your Large Multi-Modal Model at Test-Time via Non-Contrastive Visual Attribute Steeringcited by 0
  6. 2025JMMMU-Pro: Image-based Japanese Multi-discipline Multimodal Understanding Benchmark via Vibe Benchmark Constructioncited by 0
  7. 2026MM-SCALE: Grounded Multimodal Moral Reasoning via Scalar Judgment and Listwise Alignmentcited by 0
  8. 2026CrossCult-KIBench: A Benchmark for Cross-Cultural Knowledge Insertion in MLLMscited by 0
  9. 2026Causal Bias Detection in Generative Artificial Intelligencecited by 0
  10. 2026Multimodal Knowledge Edit-Scoped Generalization for Online Recursive MLLM Editingcited by 0
  11. 2026DeepBias: Adaptive In-depth Probing of Social Biases in LVLMscited by 0
  12. 2026Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Modelscited by 0
  13. 2026FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Makingcited by 0
  14. 2026VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervisioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024debiaSAE: Benchmarking and Mitigating Vision-Language Model Bias
  2. 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals
  3. 2024BIGbench: A Unified Benchmark for Evaluating Multi-dimensional Social Biases in Text-to-Image Models
  4. 2025Ask Me Again Differently: GRAS for Measuring Bias in Vision Language Models on Gender, Race, Age, and Skin Tone
  5. 2025Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessment
  6. 2024Uncovering Bias in Large Vision-Language Models with Counterfactuals
  7. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images
  8. 2025Data Matters Most: Auditing Social Bias in Contrastive Vision Language Models