mlsanityData to 7 Oct 2026

VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution

arXiv 2306.12424 ↗First posted 21 Jun 2023cs.CV cs.CLcited by 52

On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs

Abstract (arXiv)

We introduce VisoGender, a novel dataset for benchmarking gender bias in vision-language models. We focus on occupation-related biases within a hegemonic system of binary gender, inspired by Winograd and Winogender schemas, where each image is associated with a caption containing a pronoun relationship of subjects and objects in the scene. VisoGender is balanced by gender representation in professional roles, supporting bias evaluation in two ways: i) resolution bias, where we evaluate the difference between pronoun resolution accuracies for image subjects with gender presentations perceived as masculine versus feminine by human annotators and ii) retrieval bias, where we compare ratios of professionals perceived to have masculine and feminine gender presentations retrieved for a gender-neutral search query. We benchmark several state-of-the-art vision-language models and find that they demonstrate bias in resolving binary gender in complex scenes. While the direction and magnitude of gender bias depends on the task and the model being evaluated, captioning models are generally less biased than Vision-Language Encoders. Dataset and code are available at https://github.com/oxai/visogender

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 73 references that are in the map. First posted on arXiv 21 Jun 2023.

Builds on

57 of 73 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Balancing the Picture: Debiasing Vision-Language Datasets with Synthetic Contrast Sets24 May
  2. 2023Debiasing Vision-Language Models via Biased Prompts31 Jan
  3. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning22 Mar

Other references · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2022Visual Prompt Tuningcited by 1,674
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  12. 2018Datasheets for Datasetscited by 1,371
38 more
  1. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  2. 2021Ethical and social risks of harm from Language Modelscited by 957
  3. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  4. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  6. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  7. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  8. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  9. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  10. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  11. 2018Gender Bias in Coreference Resolutioncited by 438
  12. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  13. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  14. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 326
  15. 2017End-to-end Neural Coreference Resolutioncited by 240
  16. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  17. 2023Stable Bias: Analyzing Societal Representations in Diffusion Modelscited by 225
  18. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  19. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  20. 2019Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Searchcited by 137
  21. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  22. 2021Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologiescited by 128
  23. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  24. 2023Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentationcited by 100
  25. 2021Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Searchcited by 90
  26. 2021Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Developmentcited by 85
  27. 2019Toward Gender-Inclusive Coreference Resolutioncited by 84
  28. 2016Measuring Fairness in Ranked Outputscited by 81
  29. 2016Deep Reinforcement Learning for Mention-Ranking Coreference Modelscited by 63
  30. 2021Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translationcited by 62
  31. 2016Improving Coreference Resolution by Learning Entity-Level Distributed Representationscited by 50
  32. 2022Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionalitycited by 44
  33. 2023MISGENDERED: Limits of Large Language Models in Understanding Pronounscited by 43
  34. 2022Gender Artifacts in Visual Datasetscited by 38
  35. 2020Words aren't enough, their order matters: On the Robustness of Grounding Visual Referring Expressionscited by 33
  36. 2021COVR: A test-bed for Visually Grounded Compositional Generalization with real imagescited by 25
  37. 2016Learning Global Features for Coreference Resolutioncited by 20
  38. 2023Mitigating Test-Time Bias for Fair Image Retrievalcited by 20

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 20; 2025: 19; 2026: 11.

52 papers.

Sort:

By area:

  1. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  2. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  3. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  4. 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examplescited by 53
  5. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Modelscited by 44
  6. 2024GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probingcited by 31
  7. 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactualscited by 27
  8. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluationcited by 19
  9. 2024Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answeringcited by 18
  10. 2024DiffusionPID: Interpreting Diffusion via Partial Information Decompositioncited by 16
  11. 2024Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)cited by 15
  12. 2025AI Sees Your Location, But With A Bias Toward The Wealthy Worldcited by 14
38 more
  1. 2024BiasDora: Exploring Hidden Biased Associations in Vision-Language Modelscited by 12
  2. 2024Fairness and Bias Mitigation in Computer Vision: A Surveycited by 12
  3. 2025A Large Scale Analysis of Gender Biases in Text-to-Image Generative Modelscited by 12
  4. 2024A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Modelscited by 11
  5. 2025BBQ-V: Benchmarking Visual Stereotype Bias in Large Multimodal Modelscited by 10
  6. 2024Social Debiasing for Fair Multi-modal LLMscited by 8
  7. 2025VLDBench Evaluating Multimodal Disinformation with Regulatory Alignmentcited by 8
  8. 2024Nearest Neighbor Normalization Improves Multimodal Retrievalcited by 7
  9. 2024FairJob: A Real-World Dataset for Fairness in Online Systemscited by 6
  10. 2025Intrinsic Bias is Predicted by Pretraining Data and Correlates with Downstream Performance in Vision-Language Encoderscited by 6
  11. 2024What an Elegant Bridge: Multilingual LLMs are Biased Similarly in Different Languagescited by 5
  12. 2025Debiasing CLIP: Interpreting and Correcting Bias in Attention Headscited by 4
  13. 2025Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Modelscited by 4
  14. 2025Decoupling Template Bias in CLIP: Harnessing Empty Prompts for Enhanced Few-Shot Learningcited by 4
  15. 2024Evaluating Model Bias Requires Characterizing its Mistakescited by 3
  16. 2025Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessmentcited by 3
  17. 2025STATUS Bench: A Rigorous Benchmark for Evaluating Object State Understanding in Vision-Language Modelscited by 3
  18. 2025DSO: Direct Steering Optimization for Bias Mitigationcited by 3
  19. 2024Multi-Group Proportional Representation in Retrievalcited by 2
  20. 2024GABInsight: Exploring Gender-Activity Binding Bias in Vision-Language Modelscited by 2
  21. 2025From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationshipcited by 2
  22. 2025Freeze and Reveal: Exposing Modality Bias in Vision-Language Modelscited by 2
  23. 2026VEAT Quantifies Implicit Associations in Text-to-Video Generator Sora and Reveals Challenges in Bias Mitigationcited by 2
  24. 2026Cultural Counterfactuals: Evaluating Cultural Biases in Large Vision-Language Models with Counterfactual Examplescited by 2
  25. 2024debiaSAE: Benchmarking and Mitigating Vision-Language Model Biascited by 1
  26. 2025Data Matters Most: Auditing Social Bias in Contrastive Vision Language Modelscited by 1
  27. 2025Pruning the Paradox: How CLIP's Most Informative Heads Enhance Performance While Amplifying Biascited by 1
  28. 2025Synthetic History: Evaluating Visual Representations of the Past in Diffusion Modelscited by 1
  29. 2026CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Modelscited by 1
  30. 2026Vision-Language Models Suppress Female Representations Under Ambiguous Inputcited by 1
  31. 2026StylisticBias: A Few Human Visual Cues Drive Most Social Biases in MLLMscited by 1
  32. 2024Think Before You Act: A Two-Stage Framework for Mitigating Gender Bias Towards Vision-Language Taskscited by 0
  33. 2025A Bayesian account of pronoun and neopronoun acquisitioncited by 0
  34. 2025When Cars Have Stereotypes: Auditing Demographic Bias in Objects from Text-to-Image Modelscited by 0
  35. 2026Cross-Cultural Value Attribution in Large Vision-Language Modelscited by 0
  36. 2026Beyond Perception Errors: Semantic Fixation in Large Vision-Language Modelscited by 0
  37. 2026Operationalizing Fairness in Text-to-Image Models: A Survey of Bias, Fairness Audits and Mitigation Strategiescited by 0
  38. 2026DeepBias: Adaptive In-depth Probing of Social Biases in LVLMscited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Gender Bias in Vision-Language In-Context Learning
  2. 2025Breaking Language Barriers or Reinforcing Bias? A Study of Gender and Racial Disparities in Multilingual Contrastive Vision Language Models
  3. 2025Vision-Language Models display a strong gender bias
  4. 2023Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparities
  5. 2025BioPro: Towards Difference-Aware Gender Fairness for Vision-Language Models
  6. 2024Images Speak Louder than Words: Understanding and Mitigating Bias in Vision-Language Model from a Causal Mediation Perspective
  7. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images
  8. 2026ViD: Vision-Dominant Gender Bias Mitigation for Large Vision-Language Models