mlsanityData to 7 Oct 2026

Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts

arXiv 2406.17974 ↗First posted 25 Jun 2024cs.CL cs.CVcited by 15

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Large vision-language models (LVLMs) have recently achieved significant progress, demonstrating strong capabilities in open-world visual understanding. However, it is not yet clear how LVLMs address demographic biases in real life, especially the disparities across attributes such as gender, skin tone, age and race. In this paper, We empirically investigate visual fairness in several mainstream LVLMs by auditing their performance disparities across demographic attributes using public fairness benchmark datasets (e.g., FACET, UTKFace). Our fairness evaluation framework employs direct and single-choice question prompt on visual question-answering/classification tasks. Despite advancements in visual understanding, our zero-shot prompting results show that both open-source and closed-source LVLMs continue to exhibit fairness issues across different prompts and demographic groups. Furthermore, we propose a potential multi-modal Chain-of-thought (CoT) based strategy for unfairness mitigation, applicable to both open-source and closed-source LVLMs. This approach enhances transparency and offers a scalable solution for addressing fairness, providing a solid foundation for future research and practical efforts in unfairness mitigation. The dataset and code used in this study are publicly available at this GitHub Repository.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 55 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

24 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images8 Feb

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  7. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  8. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  9. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 426
  10. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  11. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  12. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 124
11 more
  1. 2023FACET: Fairness in Computer Vision Evaluation Benchmarkcited by 58
  2. 2023FFB: A Fair Fairness Benchmark for In-Processing Group Fairness Methodscited by 33
  3. 2023Efficient Multimodal Fusion via Interactive Promptingcited by 32
  4. 2022On Guiding Visual Attention with Language Specificationcited by 30
  5. 2023On the Independence of Association Bias and Empirical Fairness in Language Modelscited by 29
  6. 2022Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Surveycited by 22
  7. 2023Can Language Models Understand Physical Concepts?cited by 19
  8. 2024SQ-LLaVA: Self-Questioning for Large Vision-Language Assistantcited by 18
  9. 2024Do Large Language Models Rank Fairly? An Empirical Study on the Fairness of LLMs as Rankerscited by 17
  10. 2023Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Promptscited by 14
  11. 2023Mitigating Bias for Question Answering Models by Tracking Bias Influencecited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 3.

15 papers.

Sort:

By area:

  1. 2025Does Reasoning Introduce Bias? A Study of Social Bias Evaluation and Mitigation in LLM Reasoningcited by 20
  2. 2024Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)cited by 15
  3. 2024Social Debiasing for Fair Multi-modal LLMscited by 8
  4. 2025Building Trustworthy Multimodal AI: A Review of Fairness, Transparency, and Ethics in Vision-Language Taskscited by 7
  5. 2025Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Modelscited by 7
  6. 2025On Fairness of Task Arithmetic: The Role of Task Vectorscited by 3
  7. 2025Bias Beyond Demographics: Probing Decision Boundaries in Black-Box LVLMs via Counterfactual VQAcited by 2
  8. 2026Cultural Counterfactuals: Evaluating Cultural Biases in Large Vision-Language Models with Counterfactual Examplescited by 2
  9. 2024debiaSAE: Benchmarking and Mitigating Vision-Language Model Biascited by 1
  10. 2025Mapping User Trust in Vision Language Models: Research Landscape, Challenges, and Prospectscited by 1
  11. 2025A Stereotype Content Analysis on Color-related Social Bias in Large Vision Language Modelscited by 1
  12. 2025Interpreting Social Bias in LVLMs via Information Flow Analysis and Multi-Round Dialogue Evaluationcited by 1
3 more
  1. 2025Prompting Away Stereotypes? Evaluating Bias in Text-to-Image Models for Occupationscited by 1
  2. 2026Cross-Cultural Expert-Level Art Critique Evaluation with Vision-Language Modelscited by 1
  3. 2026Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Ask Me Again Differently: GRAS for Measuring Bias in Vision Language Models on Gender, Race, Age, and Skin Tone
  2. 2026FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making
  3. 2025Unveiling the "Fairness Seesaw": Discovering and Mitigating Gender and Race Bias in Vision-Language Models
  4. 2024GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing
  5. 2025DemoBias: An Empirical Study to Trace Demographic Biases in Vision Foundation Models
  6. 2025Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessment
  7. 2024VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model
  8. 2026Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMs