mlsanityData to 7 Oct 2026

Survey of Social Bias in Vision-Language Models

arXiv 2309.14381 ↗First posted 24 Sep 2023cs.CL cs.AIcited by 26

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

In recent years, the rapid advancement of machine learning (ML) models, particularly transformer-based pre-trained models, has revolutionized Natural Language Processing (NLP) and Computer Vision (CV) fields. However, researchers have discovered that these models can inadvertently capture and reinforce social biases present in their training datasets, leading to potential social harms, such as uneven resource allocation and unfair representation of specific social groups. Addressing these biases and ensuring fairness in artificial intelligence (AI) systems has become a critical concern in the ML community. The recent introduction of pre-trained vision-and-language (VL) models in the emerging multimodal field demands attention to the potential social biases present in these models as well. Although VL models are susceptible to social bias, there is a limited understanding compared to the extensive discussions on bias in NLP and CV. This survey aims to provide researchers with a high-level insight into the similarities and differences of social bias studies in pre-trained models across NLP, CV, and VL. By examining these perspectives, the survey aims to offer valuable guidelines on how to approach and mitigate social bias in both unimodal and multimodal settings. The findings and recommendations presented here can benefit the ML community, fostering the development of fairer and non-biased AI models in various applications and research endeavors.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 147 of its 203 references that are in the map. First posted on arXiv 24 Sep 2023.

Builds on

147 of 203 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023DeAR: Debiasing Vision-Language Models with Additive Residuals18 Mar
  2. 2023Debiasing Vision-Language Models via Biased Prompts31 Jan
  3. 2022Counterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Models3 Jul
  4. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning22 Mar

Other references · 143

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2017Mask R-CNNcited by 7,259
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2016Equality of Opportunity in Supervised Learningcited by 2,000
38 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  3. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  6. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  12. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  13. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  14. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  15. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  16. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  17. 2017Counterfactual Fairnesscited by 802
  18. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  19. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  20. 2018Mitigating Unwanted Biases with Adversarial Learningcited by 663
  21. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  22. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  23. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  24. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  25. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  26. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  27. 2018Gender Bias in Coreference Resolutioncited by 438
  28. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 426
  29. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  30. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  31. 2017On Fairness and Calibrationcited by 406
  32. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  33. 2017Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairnesscited by 381
  34. 2018Learning Adversarially Fair and Transferable Representationscited by 353
  35. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  36. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  37. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  38. 2022Scaling Language-Image Pre-training via Maskingcited by 317

50 most cited of 143 All 143 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 13; 2026: 4.

26 papers.

Sort:

By area:

  1. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  2. 2025Vision Language Models are Biasedcited by 63
  3. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  4. 2026Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluationscited by 36
  5. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  6. 2024GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probingcited by 31
  7. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluationcited by 19
  8. 2024Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)cited by 15
  9. 2024The Pursuit of Fairness in Artificial Intelligence Models: A Surveycited by 13
  10. 2024Social Perception of Faces in a Vision-Language Modelcited by 10
  11. 2024Gender Bias in Text-to-Video Generation Models: A case study of Soracited by 8
  12. 2025Building Trustworthy Multimodal AI: A Review of Fairness, Transparency, and Ethics in Vision-Language Taskscited by 7
14 more
  1. 2026Interpretable Debiasing of Vision-Language Models for Social Fairnesscited by 6
  2. 2025Vision-Language Models display a strong gender biascited by 5
  3. 2025FAIR-SIGHT: Fairness Assurance in Image Recognition via Simultaneous Conformal Thresholding and Dynamic Output Repaircited by 3
  4. 2025Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessmentcited by 3
  5. 2025DSO: Direct Steering Optimization for Bias Mitigationcited by 3
  6. 2024GABInsight: Exploring Gender-Activity Binding Bias in Vision-Language Modelscited by 2
  7. 2025Beyond Conventional Transformers: The Medical X-ray Attention (MXA) Block for Improved Multi-Label Diagnosis Using Knowledge Distillationcited by 2
  8. 2025Bias Beyond Demographics: Probing Decision Boundaries in Black-Box LVLMs via Counterfactual VQAcited by 2
  9. 2024debiaSAE: Benchmarking and Mitigating Vision-Language Model Biascited by 1
  10. 2025MOSCARD -- Causal Reasoning and De-confounding for Multimodal Opportunistic Screening of Cardiovascular Adverse Eventscited by 1
  11. 2024Data Quality in Edge Machine Learning: A State-of-the-Art Surveycited by 0
  12. 2025On the Reproducibility of "FairCLIP: Harnessing Fairness in Vision-Language Learning''cited by 0
  13. 2026When Visual Evidence is Ambiguous: Pareidolia as a Diagnostic Probe for Vision Modelscited by 0
  14. 2026Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models
  2. 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals
  3. 2025VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models
  4. 2024Social Debiasing for Fair Multi-modal LLMs
  5. 2024Identifying Implicit Social Biases in Vision-Language Models
  6. 2024VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model
  7. 2023A Multidimensional Analysis of Social Biases in Vision Transformers
  8. 2024Uncovering Bias in Large Vision-Language Models with Counterfactuals