mlsanityData to 7 Oct 2026

MultiModal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision Language Models

arXiv 2303.12734 ↗First posted 16 Mar 2023cs.CV cs.CL cs.LGcited by 23

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Recent breakthroughs in self supervised training have led to a new class of pretrained vision language models. While there have been investigations of bias in multimodal models, they have mostly focused on gender and racial bias, giving much less attention to other relevant groups, such as minorities with regard to religion, nationality, sexual orientation, or disabilities. This is mainly due to lack of suitable benchmarks for such groups. We seek to address this gap by providing a visual and textual bias benchmark called MMBias, consisting of around 3,800 images and phrases covering 14 population subgroups. We utilize this dataset to assess bias in several prominent self supervised multimodal models, including CLIP, ALBEF, and ViLT. Our results show that these models demonstrate meaningful bias favoring certain groups. Finally, we introduce a debiasing method designed specifically for such large pre-trained models that can be applied as a post-processing step to mitigate bias, while preserving the remaining accuracy of the model.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 41 references that are in the map. First posted on arXiv 16 Mar 2023.

Builds on

22 of 41 references.

References · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  6. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  7. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  8. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  9. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  10. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  11. 2020Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biasescited by 167
  12. 2019Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddingscited by 146
10 more
  1. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  2. 2020Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biasescited by 107
  3. 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
  4. 2021Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?cited by 85
  5. 2020Measuring Social Biases in Grounded Vision and Language Embeddingscited by 61
  6. 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Modelscited by 52
  7. 2019Exposing and Correcting the Gender Bias in Image Captioning Datasets and Modelscited by 36
  8. 2021Assessing Multilingual Fairness in Pre-trained Multimodal Representationscited by 35
  9. 2020Sentiment Analysis: Automatically Detecting Valence, Emotions, and Other Affectual States from Textcited by 26
  10. 2022A study on the distribution of social biases in self-supervised learning visual modelscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 10; 2026: 4.

23 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  3. 2024On the Challenges and Opportunities in Generative AIcited by 31
  4. 2024GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probingcited by 31
  5. 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactualscited by 27
  6. 2024MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platformscited by 26
  7. 2024VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Modelcited by 26
  8. 2024Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)cited by 15
  9. 2024Social Debiasing for Fair Multi-modal LLMscited by 8
  10. 2026Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusionscited by 8
  11. 2025Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Modelscited by 7
  12. 2024FairPIVARA: Reducing and Assessing Biases in CLIP-Based Multimodal Modelscited by 4
11 more
  1. 2025Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Modelscited by 4
  2. 2025DSO: Direct Steering Optimization for Bias Mitigationcited by 3
  3. 2025Cultural Awareness in Vision-Language Models: A Cross-Country Explorationcited by 2
  4. 2026Cultural Counterfactuals: Evaluating Cultural Biases in Large Vision-Language Models with Counterfactual Examplescited by 2
  5. 2025Data Matters Most: Auditing Social Bias in Contrastive Vision Language Modelscited by 1
  6. 2025FairSSL: Fair Multimodal Self-Supervised Learningcited by 1
  7. 2025Surgeons Are Indian Males and Speech Therapists Are White Females: Auditing Biases in Vision-Language Models for Healthcare Professionalscited by 1
  8. 2024Think Before You Act: A Two-Stage Framework for Mitigating Gender Bias Towards Vision-Language Taskscited by 0
  9. 2025MoralCLIP: Contrastive Alignment of Vision-and-Language Representations with Moral Foundations Theorycited by 0
  10. 2026IndicFairFace: Balanced Indian Face Dataset for Auditing and Mitigating Geographical Bias in Vision-Language Modelscited by 0
  11. 2026Addressing Overcommitment in the Reasoning of Gendered Economic Memes under Multimodal Ambiguitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Freeze and Reveal: Exposing Modality Bias in Vision-Language Models
  2. 2024A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models
  3. 2024Identifying Implicit Social Biases in Vision-Language Models
  4. 2024debiaSAE: Benchmarking and Mitigating Vision-Language Model Bias
  5. 2025BBQ-V: Benchmarking Visual Stereotype Bias in Large Multimodal Models
  6. 2025Breaking Language Barriers or Reinforcing Bias? A Study of Gender and Racial Disparities in Multilingual Contrastive Vision Language Models
  7. 2025Ask Me Again Differently: GRAS for Measuring Bias in Vision Language Models on Gender, Race, Age, and Skin Tone
  8. 2025VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models