# VisBias: Measuring Explicit and Implicit Social Biases in Vision Language Models

arXiv: https://arxiv.org/abs/2503.07575 · HTML: https://mlsanity.com/paper/2503.07575/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 57 references that are in the map.
- First posted on arXiv 10 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Social bias and fairness in VLMs](https://mlsanity.com/mllm/social-bias-and-fairness-in-vlms/)

## Builds on (most cited first, 13 of 13)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision](https://mlsanity.com/paper/2310.18689/) (2023)
- [MultiMath: Bridging Visual and Mathematical Reasoning for Large Language Models](https://mlsanity.com/paper/2409.00147/) (2024)
- [Ocean-OCR: Towards General OCR Application via a Vision-Language Model](https://mlsanity.com/paper/2501.15558/) (2025)
- [MathGLM-Vision: Solving Mathematical Problems with Multi-Modal Large Language Model](https://mlsanity.com/paper/2409.13729/) (2024)
- [Multimodal Foundation Models Exploit Text to Make Medical Image Predictions](https://mlsanity.com/paper/2311.05591/) (2023)
- [BIGbench: A Unified Benchmark for Evaluating Multi-dimensional Social Biases in Text-to-Image Models](https://mlsanity.com/paper/2407.15240/) (2024)
- [Fairness Testing of Large Language Models in Role-Playing](https://mlsanity.com/paper/2411.00585/) (2024)
- [FairCoder: Probing LLM Bias in High-Stakes Decision Making via Coding Tasks](https://mlsanity.com/paper/2501.05396/) (2025)

## Built on by (most cited first, 14 of 14)

- [The Chameleon's Limit: Investigating Persona Collapse and Homogenization in Large Language Models](https://mlsanity.com/paper/2604.24698/) (2026)
- [VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models](https://mlsanity.com/paper/2505.22897/) (2025)
- [DSO: Direct Steering Optimization for Bias Mitigation](https://mlsanity.com/paper/2512.15926/) (2025)
- [From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationship](https://mlsanity.com/paper/2506.23101/) (2025)
- [Bias Beyond Demographics: Probing Decision Boundaries in Black-Box LVLMs via Counterfactual VQA](https://mlsanity.com/paper/2508.03079/) (2025)
- [Large Language Models Develop Novel Social Biases Through Adaptive Exploration](https://mlsanity.com/paper/2511.06148/) (2025)
- [Identifying Ethical Biases in Action Recognition Models](https://mlsanity.com/paper/2604.17971/) (2026)
- [Enhancing Lightweight Vision Language Models through Group Competitive Learning for Socially Compliant Navigation](https://mlsanity.com/paper/2603.11447/) (2026)
- [EgoGroups: A Benchmark For Detecting Social Groups of People in the Wild](https://mlsanity.com/paper/2603.22249/) (2026)
- [Demographic and Linguistic Bias Evaluation in Omnimodal Language Models](https://mlsanity.com/paper/2604.10014/) (2026)
- [Artificial Intolerance: Stigmatizing Language in Clinical Documentation Skews Large Language Model Decision-Making](https://mlsanity.com/paper/2605.17228/) (2026)
- [Mitigating Bias in Large Vision-Language Models via Counterfactual Ensemble Decoding](https://mlsanity.com/paper/2608.21415/) (2026)
- [Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models](https://mlsanity.com/paper/2608.29590/) (2026)
- [FairLens: Benchmarking Fairness in Vision-Language Models for High-Stakes Decision-Making](https://mlsanity.com/paper/2609.01691/) (2026)

## Abstract (arXiv)

This research investigates both explicit and implicit social biases exhibited by Vision-Language Models (VLMs). The key distinction between these bias types lies in the level of awareness: explicit bias refers to conscious, intentional biases, while implicit bias operates subconsciously. To analyze explicit bias, we directly pose questions to VLMs related to gender and racial differences: (1) Multiple-choice questions based on a given image (e.g., "What is the education level of the person in the image?") (2) Yes-No comparisons using two images (e.g., "Is the person in the first image more educated than the person in the second image?") For implicit bias, we design tasks where VLMs assist users but reveal biases through their responses: (1) Image description tasks: Models are asked to describe individuals in images, and we analyze disparities in textual cues across demographic groups. (2) Form completion tasks: Models draft a personal information collection form with 20 attributes, and we examine correlations among selected attributes for potential biases. We evaluate Gemini-1.5, GPT-4V, GPT-4o, LLaMA-3.2-Vision and LLaVA-v1.6. Our code and data are publicly available at https://github.com/uscnlp-lime/VisBias.