mlsanityData to 7 Oct 2026

Counterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Models

arXiv 2207.01056 ↗First posted 3 Jul 2022cs.CV cs.AIcited by 28

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Vision-Language Pre-training (VLP) models have achieved state-of-the-art performance in numerous cross-modal tasks. Since they are optimized to capture the statistical properties of intra- and inter-modality, there remains risk to learn social biases presented in the data as well. In this work, we (1) introduce a counterfactual-based bias measurement CounterBias to quantify the social bias in VLP models by comparing the [MASK]ed prediction probabilities of factual and counterfactual samples; (2) construct a novel VL-Bias dataset including 24K image-text pairs for measuring gender bias in VLP models, from which we observed that significant gender bias is prevalent in VLP models; and (3) propose a VLP debiasing method FairVLP to minimize the difference in the [MASK]ed prediction probabilities between factual and counterfactual image-text pairs for VLP debiasing. Although CounterBias and FairVLP focus on social bias, they are generalizable to serve as tools and provide new insights to probe and regularize more knowledge in VLP models.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 45 references that are in the map. First posted on arXiv 3 Jul 2022.

Builds on

36 of 45 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2016Adversarial examples in the physical worldcited by 1,762
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
24 more
  1. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  2. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  3. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  4. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  5. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  6. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  7. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 426
  8. 2020PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Modelscited by 358
  9. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  10. 2019Measuring Bias in Contextualized Word Representationscited by 275
  11. 2020Measuring and Reducing Gendered Correlations in Pre-trained Modelscited by 222
  12. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  13. 2019Gender Bias in Contextualized Word Embeddingscited by 203
  14. 2019Fair Generative Modeling via Weak Supervisioncited by 114
  15. 2019Bias Correction of Learned Generative Models using Likelihood-Free Importance Weightingcited by 70
  16. 2020Measuring Social Biases in Grounded Vision and Language Embeddingscited by 61
  17. 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Modelscited by 52
  18. 2020DeVLBert: Learning Deconfounded Visio-Linguistic Representationscited by 42
  19. 2019Automatic Testing and Improvement of Machine Translationcited by 35
  20. 2020Towards Accuracy-Fairness Paradox: Adversarial Example-based Data Augmentation for Visual Debiasingcited by 32
  21. 2021ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integrationcited by 31
  22. 2020VIVO: Visual Vocabulary Pre-Training for Novel Object Captioningcited by 30
  23. 2020Imperfect ImaGANation: Implications of GANs Exacerbating Biases on Facial Data Augmentation and Snapchat Selfie Lensescited by 27
  24. 2021CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoisingcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 13; 2025: 4; 2026: 3.

28 papers.

Sort:

By area:

  1. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  2. 2025CF-VLM:CounterFactual Vision-Language Fine-tuningcited by 57
  3. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Imagescited by 55
  4. 2023Fair Text-to-Image Diffusion via Fair Mappingcited by 31
  5. 2024GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probingcited by 31
  6. 2023Survey of Social Bias in Vision-Language Modelscited by 26
  7. 2023Gender Biases in Automatic Evaluation Metrics for Image Captioningcited by 24
  8. 2024Fairness and Bias in Multimodal AI: A Surveycited by 19
  9. 2023Model-Agnostic Gender Debiased Image Captioningcited by 17
  10. 2024New Job, New Gender? Measuring the Social Bias in Image Generation Modelscited by 15
  11. 2024Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)cited by 15
  12. 2025AI Sees Your Location, But With A Bias Toward The Wealthy Worldcited by 14
16 more
  1. 2024Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspectivecited by 13
  2. 2024FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplicationcited by 11
  3. 2023Adventures of Trustworthy Vision-Language Models: A Surveycited by 10
  4. 2024The Bias of Harmful Label Associations in Vision-Language Modelscited by 5
  5. 2024Aligning Visual Contrastive learning models via Preference Optimizationcited by 4
  6. 2025Self-Adaptive Cognitive Debiasing for Large Language Models in Decision-Makingcited by 4
  7. 2024They're All Doctors: Synthesizing Diverse Counterfactuals to Mitigate Associative Biascited by 3
  8. 2023Benign Shortcut for Debiasing: Fair Visual Recognition via Intervention with Shortcut Featurescited by 2
  9. 2023Semantically Grounded QFormer for Efficient Vision Language Understandingcited by 2
  10. 2024GABInsight: Exploring Gender-Activity Binding Bias in Vision-Language Modelscited by 2
  11. 2025TPC: Cross-Temporal Prediction Connection for Vision-Language Model Hallucination Reductioncited by 2
  12. 2026Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photoscited by 2
  13. 2022Fair Visual Recognition via Intervention with Proxy Featurescited by 0
  14. 2024Think Before You Act: A Two-Stage Framework for Mitigating Gender Bias Towards Vision-Language Taskscited by 0
  15. 2026SDGBiasBench: Benchmarking and Mitigating Vision--Language Models' Biases in Sustainable Development Goalscited by 0
  16. 2026DeepBias: Adaptive In-depth Probing of Social Biases in LVLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Probing Intersectional Biases in Vision-Language Models with Counterfactual Examples
  2. 2025Freeze and Reveal: Exposing Modality Bias in Vision-Language Models
  3. 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals
  4. 2024Uncovering Bias in Large Vision-Language Models with Counterfactuals
  5. 2024A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models
  6. 2025Data Matters Most: Auditing Social Bias in Contrastive Vision Language Models
  7. 2023MultiModal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision Language Models
  8. 2024Social Debiasing for Fair Multi-modal LLMs