# From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationship

arXiv: https://arxiv.org/abs/2506.23101 · HTML: https://mlsanity.com/paper/2506.23101/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 36 of its 43 references that are in the map.
- First posted on arXiv 29 Jun 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Social bias and fairness in VLMs](https://mlsanity.com/mllm/social-bias-and-fairness-in-vlms/)

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- ["Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters](https://mlsanity.com/paper/2310.09219/) (2023)
- [PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding](https://mlsanity.com/paper/2501.16411/) (2025)
- [Social Biases through the Text-to-Image Generation Lens](https://mlsanity.com/paper/2304.06034/) (2023)
- [Understanding and Evaluating Racial Biases in Image Captioning](https://mlsanity.com/paper/2106.08503/) (2021)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigation](https://mlsanity.com/paper/2404.01030/) (2024)
- [Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images](https://mlsanity.com/paper/2402.05779/) (2024)
- [SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples](https://mlsanity.com/paper/2312.00825/) (2023)
- [Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2104.08666/) (2021)
- [VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution](https://mlsanity.com/paper/2306.12424/) (2023)
- [BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models](https://mlsanity.com/paper/2312.02896/) (2023)
- [MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models](https://mlsanity.com/paper/2406.07594/) (2024)
- [Img2Loc: Revisiting Image Geolocalization using Multi-modality Foundation Models and Image-based Retrieval-Augmented Generation](https://mlsanity.com/paper/2403.19584/) (2024)
- [GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing](https://mlsanity.com/paper/2407.00600/) (2024)
- [Gender Biases in Automatic Evaluation Metrics for Image Captioning](https://mlsanity.com/paper/2305.14711/) (2023)
- [Benchmarking Large Multimodal Models against Common Corruptions](https://mlsanity.com/paper/2401.11943/) (2024)
- [Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Models](https://mlsanity.com/paper/2304.13855/) (2023)
- [Gender Bias in Decision-Making with Large Language Models: A Study of Relationship Conflicts](https://mlsanity.com/paper/2410.11084/) (2024)
- [Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)](https://mlsanity.com/paper/2410.19314/) (2024)
- [VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models](https://mlsanity.com/paper/2503.07575/) (2025)
- [A Large Scale Analysis of Gender Biases in Text-to-Image Generative Models](https://mlsanity.com/paper/2503.23398/) (2025)

## Built on by (most cited first, 2 of 2)

- [When Understanding Becomes a Risk: Authenticity and Safety Risks in the Emerging Image Generation Paradigm](https://mlsanity.com/paper/2603.24079/) (2026)
- [Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models](https://mlsanity.com/paper/2608.29590/) (2026)

## Abstract (arXiv)

Ensuring the trustworthiness of Multimodal Large Language Models (MLLMs) requires rigorous evaluation of gender bias, particularly in socially sensitive domains. Existing benchmarks primarily assess bias in isolated contexts and are less suited to capturing biases that emerge through interpersonal interactions. We introduce Genres, a benchmark for evaluating gender bias in MLLMs through the lens of social relationships in narrative generation. Grounded in Fiske's relational models theory, Genres comprises 1,440 multimodal narrative elicitation pairs spanning four fundamental relationship types, diverse social scenarios, and three age groups. Its dual-character generation tasks elicit socially situated interpersonal narratives and enable fine-grained assessment across multiple dimensions of gender bias. Experiments on six recent MLLMs reveal persistent yet strongly relationship- and model-dependent gender disparities, with some patterns becoming more pronounced in interpersonal settings. We further evaluate two complementary generation-stage mitigation strategies. Our findings demonstrate the value of relationship-aware evaluation for uncovering subtle, interaction-driven gender bias in MLLMs.