mlsanityData to 7 Oct 2026

Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models

arXiv 2104.08666 ↗First posted 18 Apr 2021cs.CLcited by 52

Not on the map. Why?

Abstract (arXiv)

Numerous works have analyzed biases in vision and pre-trained language models individually - however, less attention has been paid to how these biases interact in multimodal settings. This work extends text-based bias analysis methods to investigate multimodal language models, and analyzes intra- and inter-modality associations and biases learned by these models. Specifically, we demonstrate that VL-BERT (Su et al., 2020) exhibits gender biases, often preferring to reinforce a stereotype over faithfully describing the visual scene. We demonstrate these findings on a controlled case-study and extend them for a larger set of stereotypically gendered entities.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 24 of its 31 references that are in the map. First posted on arXiv 18 Apr 2021.

Builds on

24 of 31 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  8. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  11. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  12. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
12 more
  1. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  2. 2018Gender Bias in Coreference Resolutioncited by 438
  3. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  4. 2016Visual Dialogcited by 377
  5. 2019Measuring Bias in Contextualized Word Representationscited by 275
  6. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  7. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  8. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 222
  9. 2019Gender Bias in Contextualized Word Embeddingscited by 203
  10. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  11. 2018Overcoming Language Priors in Visual Question Answering with Adversarial Regularizationcited by 104
  12. 2019Exposing and Correcting the Gender Bias in Image Captioning Datasets and Modelscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 9; 2023: 13; 2024: 19; 2025: 7; 2026: 2.

52 papers.

Sort:

By area:

  1. 2024Capabilities of Gemini Models in Medicinecited by 231
  2. 2022DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Modelscited by 230
  3. 2024An Introduction to Vision-Language Modelingcited by 175
  4. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  5. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  6. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  7. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  8. 2023Uncurated Image-Text Datasets: Shedding Light on Demographic Biascited by 51
  9. 2021Assessing Multilingual Fairness in Pre-trained Multimodal Representationscited by 35
  10. 2023Overwriting Pretrained Bias with Finetuning Datacited by 32
  11. 2022The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarkscited by 31
  12. 2022Counterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Modelscited by 28
38 more
  1. 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactualscited by 27
  2. 2024Identifying Implicit Social Biases in Vision-Language Modelscited by 27
  3. 2023Survey of Social Bias in Vision-Language Modelscited by 26
  4. 2024Would Deep Generative Models Amplify Bias in Future Models?cited by 24
  5. 2023MultiModal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision Language Modelscited by 23
  6. 2024SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIPcited by 19
  7. 2023Model-Agnostic Gender Debiased Image Captioningcited by 17
  8. 2024BendVLM: Test-Time Debiasing of Vision-Language Embeddingscited by 17
  9. 2024DiffusionPID: Interpreting Diffusion via Partial Information Decompositioncited by 16
  10. 2024Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)cited by 15
  11. 2024Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspectivecited by 13
  12. 2024A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Modelscited by 11
  13. 2024FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplicationcited by 11
  14. 2023Adventures of Trustworthy Vision-Language Models: A Surveycited by 10
  15. 2023DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversitycited by 9
  16. 2024Analyzing Quality, Bias, and Performance in Text-to-Image Generative Modelscited by 9
  17. 2023VLSlice: Interactive Vision-and-Language Slice Discoverycited by 7
  18. 2024Generating Contextually-Relevant Navigation Instructions for Blind and Low Vision Peoplecited by 7
  19. 2022What's in a Caption? Dataset-Specific Linguistic Diversity and Its Effect on Visual Description Models and Metricscited by 6
  20. 2023Run Like a Girl! Sports-Related Gender Bias in Language and Visioncited by 6
  21. 2023Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Modelscited by 6
  22. 2025Bias in Gender Bias Benchmarks: How Spurious Features Distort Evaluationcited by 5
  23. 2022Manifestations of Xenophobia in AI Systemscited by 4
  24. 2023Which One? Leveraging Context Between Objects and Multiple Views for Language Groundingcited by 3
  25. 2024Streaming Detection of Queried Event Startcited by 3
  26. 2025Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessmentcited by 3
  27. 2022Vision-and-Language Pretrainingcited by 2
  28. 2024GABInsight: Exploring Gender-Activity Binding Bias in Vision-Language Modelscited by 2
  29. 2025From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationshipcited by 2
  30. 2022A Multibias-mitigated and Sentiment Knowledge Enriched Transformer for Debiasing in Multimodal Conversational Emotion Recognitioncited by 1
  31. 2023Multitask Learning Can Improve Worst-Group Outcomescited by 1
  32. 2024More is Less? A Simulation-Based Approach to Dynamic Interactions between Biases in Multimodal Modelscited by 1
  33. 2025MASS: Overcoming Language Bias in Image-Text Matchingcited by 1
  34. 2025ASCenD-BDS: Adaptable, Stochastic and Context-aware framework for Detection of Bias, Discrimination and Stereotypingcited by 1
  35. 2025Using LLMs as prompt modifier to avoid biases in AI image generatorscited by 1
  36. 2024Foundations of Multisensory Artificial Intelligencecited by 0
  37. 2024Capturing Bias Diversity in LLMscited by 0
  38. 2025Multimodal Political Bias Identification and Neutralizationcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Freeze and Reveal: Exposing Modality Bias in Vision-Language Models
  2. 2026Gender Bias in Vision-Language In-Context Learning
  3. 2024Uncovering Bias in Large Vision-Language Models with Counterfactuals
  4. 2024More Distinctively Black and Feminine Faces Lead to Increased Stereotyping in Vision-Language Models
  5. 2026When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Models
  6. 2023Controlling for Stereotypes in Multimodal Language Model Evaluation
  7. 2025VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models
  8. 2025Breaking Language Barriers or Reinforcing Bias? A Study of Gender and Racial Disparities in Multilingual Contrastive Vision Language Models