mlsanityData to 14 Oct 2026

GABInsight: Exploring Gender-Activity Binding Bias in Vision-Language Models

arXiv 2407.21001 ↗First posted 30 Jul 2024cs.CV cs.AI cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

Vision-language models (VLMs) are intensively used in many downstream tasks, including those requiring assessments of individuals appearing in the images. While VLMs perform well in simple single-person scenarios, in real-world applications, we often face complex situations in which there are persons of different genders doing different activities. We show that in such cases, VLMs are biased towards identifying the individual with the expected gender (according to ingrained gender stereotypes in the model or other forms of sample selection bias) as the performer of the activity. We refer to this bias in associating an activity with the gender of its actual performer in an image or text as the Gender-Activity Binding (GAB) bias and analyze how this bias is internalized in VLMs. To assess this bias, we have introduced the GAB dataset with approximately 5500 AI-generated images that represent a variety of activities, addressing the scarcity of real-world images for some scenarios. To have extensive quality control, the generated images are evaluated for their diversity, quality, and realism. We have tested 12 renowned pre-trained VLMs on this dataset in the context of text-to-image and image-to-text retrieval to measure the effect of this bias on their predictions. Additionally, we have carried out supplementary experiments to quantify the bias in VLMs' text encoders and to evaluate VLMs' capability to recognize activities. Our experiments indicate that VLMs experience an average performance decline of about 13.2% when confronted with gender-activity binding bias.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 45 references that are in the map. First posted on arXiv 30 Jul 2024.

Builds on

32 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Survey of Social Bias in Vision-Language Models24 Sep
  2. 2023VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution21 Jun
  3. 2023DeAR: Debiasing Vision-Language Models with Additive Residuals18 Mar
  4. 2022Counterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Models3 Jul
  5. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning22 Mar

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
15 more
  1. 2021Grounded Language-Image Pre-trainingcited by 1,156
  2. 2016Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networkscited by 903
  3. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  4. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  6. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  7. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  8. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  9. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  10. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  11. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  12. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 114
  13. 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
  14. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 85
  15. 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Modelscited by 52

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Building Trustworthy Multimodal AI: A Review of Fairness, Transparency, and Ethics in Vision-Language Taskscited by 7
  2. 2025Representation-Level Counterfactual Calibration for Debiased Zero-Shot Recognitioncited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models
  2. 2025Bias in Gender Bias Benchmarks: How Spurious Features Distort Evaluation
  3. 2024VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model
  4. 2024Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspective
  5. 2024GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing
  6. 2025Ask Me Again Differently: GRAS for Measuring Bias in Vision Language Models on Gender, Race, Age, and Skin Tone
  7. 2026Gender Bias in Vision-Language In-Context Learning
  8. 2024Think Before You Act: A Two-Stage Framework for Mitigating Gender Bias Towards Vision-Language Tasks