From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationship
On the map: Multimodal LLMs / vision-language models → Social bias and fairness in VLMs
Abstract (arXiv)
Ensuring the trustworthiness of Multimodal Large Language Models (MLLMs) requires rigorous evaluation of gender bias, particularly in socially sensitive domains. Existing benchmarks primarily assess bias in isolated contexts and are less suited to capturing biases that emerge through interpersonal interactions. We introduce Genres, a benchmark for evaluating gender bias in MLLMs through the lens of social relationships in narrative generation. Grounded in Fiske's relational models theory, Genres comprises 1,440 multimodal narrative elicitation pairs spanning four fundamental relationship types, diverse social scenarios, and three age groups. Its dual-character generation tasks elicit socially situated interpersonal narratives and enable fine-grained assessment across multiple dimensions of gender bias. Experiments on six recent MLLMs reveal persistent yet strongly relationship- and model-dependent gender disparities, with some patterns becoming more pronounced in interpersonal settings. We further evaluate two complementary generation-stage mitigation strategies. Our findings demonstrate the value of relationship-aware evaluation for uncovering subtle, interaction-driven gender bias in MLLMs.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 43 references that are in the map. First posted on arXiv 29 Jun 2025.
Builds on
36 of 43 references.
Closest ancestors · 8
Same direction, newest first
- 2025VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models10 Mar
- 2025BBQ-V: Benchmarking Visual Stereotype Bias in Large Multimodal Models12 Feb
- 2024BiasDora: Exploring Hidden Biased Associations in Vision-Language Models2 Jul
- 2024GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing30 Jun
- 2024Uncovering Bias in Large Vision-Language Models with Counterfactuals29 Mar
- 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images8 Feb
- 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples30 Nov
- 2023VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution21 Jun
Other references · 28
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2023GPT-4 Technical Reportcited by 18,396
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2024GPT-4o System Cardcited by 4,588
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
- 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
- 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
- 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
- 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 199
- 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 160
- 2023Social Biases through the Text-to-Image Generation Lenscited by 127
16 more
- 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
- 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Surveycited by 76
- 2024Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigationcited by 66
- 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Modelscited by 52
- 2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Modelscited by 41
- 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
- 2024Img2Loc: Revisiting Image Geolocalization using Multi-modality Foundation Models and Image-based Retrieval-Augmented Generationcited by 32
- 2023Gender Biases in Automatic Evaluation Metrics for Image Captioningcited by 24
- 2024Benchmarking Large Multimodal Models against Common Corruptionscited by 24
- 2023Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Modelscited by 21
- 2024Gender Bias in Decision-Making with Large Language Models: A Study of Relationship Conflictscited by 15
- 2024Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)cited by 15
- 2025A Large Scale Analysis of Gender Biases in Text-to-Image Generative Modelscited by 14
- 2024Gender Bias in LLM-generated Interview Responsescited by 10
- 2023A Multidimensional Analysis of Social Biases in Vision Transformerscited by 8
- 2025More Women, Same Stereotypes: Unpacking the Gender Bias Paradox in Large Language Modelscited by 5
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2023Gender bias and stereotypes in Large Language Models
- 2024The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models
- 2025EuroGEST: Investigating gender stereotypes in multilingual language models
- 2026Alignment Reduces Expressed but Not Encoded Gender Bias: A Unified Framework and Study
- 2026Invisible Influences: Investigating Implicit Intersectional Biases through Persona Engineering in Large Language Models
- 2026SPAGBias: Uncovering and Tracing Structured Spatial Gender Bias in Large Language Models
- 2025Mitigating Gender Bias via Fostering Exploratory Thinking in LLMs
- 2024Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals