mlsanityData to 14 Oct 2026

Controlling for Stereotypes in Multimodal Language Model Evaluation

arXiv 2302.01582 ↗First posted 3 Feb 2023cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

We propose a methodology and design two benchmark sets for measuring to what extent language-and-vision language models use the visual signal in the presence or absence of stereotypes. The first benchmark is designed to test for stereotypical colors of common objects, while the second benchmark considers gender stereotypes. The key idea is to compare predictions when the image conforms to the stereotype to predictions when it does not. Our results show that there is significant variation among multimodal models: the recent Transformer-based FLAVA seems to be more sensitive to the choice of image and less affected by stereotypes than older CNN-based models such as VisualBERT and LXMERT. This effect is more discernible in this type of controlled setting than in traditional evaluations where we do not know whether the model relied on the stereotype or the visual signal.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 28 references that are in the map. First posted on arXiv 3 Feb 2023.

Builds on

24 of 28 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2019Language Models as Knowledge Bases?cited by 1,936
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  10. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  12. 2019How Can We Know What Language Models Know?cited by 959
12 more
  1. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  2. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  3. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  4. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 137
  5. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 135
  6. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  7. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 72
  8. 2020Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting taskscited by 34
  9. 2021Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Modelscited by 18
  10. 2021Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?cited by 18
  11. 2022How to Adapt Pre-trained Vision-and-Language Models to a Text-only Input?cited by 4
  12. 2022What do Models Learn From Training on More Than Text? Measuring Visual Commonsense Knowledgecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023MultiModal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision Language Models
  2. 2025A Stereotype Content Analysis on Color-related Social Bias in Large Vision Language Models
  3. 2025Breaking Language Barriers or Reinforcing Bias? A Study of Gender and Racial Disparities in Multilingual Contrastive Vision Language Models
  4. 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models
  5. 2025Freeze and Reveal: Exposing Modality Bias in Vision-Language Models
  6. 2025BBQ-V: Benchmarking Visual Stereotype Bias in Large Multimodal Models
  7. 2025On Fairness of Unified Multimodal Large Language Model for Image Generation
  8. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias