# A Stereotype Content Analysis on Color-related Social Bias in Large Vision Language Models

arXiv: https://arxiv.org/abs/2505.20901 · HTML: https://mlsanity.com/paper/2505.20901/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 24 of its 52 references that are in the map.
- First posted on arXiv 27 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Social bias and fairness in VLMs](https://mlsanity.com/mllm/social-bias-and-fairness-in-vlms/)

## Builds on (most cited first, 24 of 24)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Debiasing Vision-Language Models via Biased Prompts](https://mlsanity.com/paper/2302.00070/) (2023)
- [Social Biases through the Text-to-Image Generation Lens](https://mlsanity.com/paper/2304.06034/) (2023)
- [Generative Language Models Exhibit Social Identity Biases](https://mlsanity.com/paper/2310.15819/) (2023)
- [VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment](https://mlsanity.com/paper/2410.09421/) (2024)
- [Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images](https://mlsanity.com/paper/2402.05779/) (2024)
- [SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples](https://mlsanity.com/paper/2312.00825/) (2023)
- [Understanding the Impact of Negative Prompts: When and How Do They Take Effect?](https://mlsanity.com/paper/2406.02965/) (2024)
- [Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgery](https://mlsanity.com/paper/2405.10948/) (2024)
- [Uncovering Bias in Large Vision-Language Models at Scale with Counterfactuals](https://mlsanity.com/paper/2405.20152/) (2024)
- [ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness](https://mlsanity.com/paper/2504.10514/) (2025)
- [Gender Biases in Automatic Evaluation Metrics for Image Captioning](https://mlsanity.com/paper/2305.14711/) (2023)
- [Understanding and Countering Stereotypes: A Computational Approach to the Stereotype Content Model](https://mlsanity.com/paper/2106.02596/) (2021)
- [textttModSCAN: Measuring Stereotypical Bias in Large Vision-Language Models from Vision and Language Modalities](https://mlsanity.com/paper/2410.06967/) (2024)
- [New Job, New Gender? Measuring the Social Bias in Image Generation Models](https://mlsanity.com/paper/2401.00763/) (2024)
- [Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Prompts](https://mlsanity.com/paper/2406.17974/) (2024)
- [Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images](https://mlsanity.com/paper/2502.13928/) (2025)
- [A Unified Framework and Dataset for Assessing Societal Bias in Vision-Language Models](https://mlsanity.com/paper/2402.13636/) (2024)
- [Color Channel Perturbation Attacks for Fooling Convolutional Neural Networks and A Defense Against Such Attacks](https://mlsanity.com/paper/2012.14456/) (2020)
- [When Tom Eats Kimchi: Evaluating Cultural Bias of Multimodal Large Language Models in Cultural Mixture Contexts](https://mlsanity.com/paper/2503.16826/) (2025)

## Built on by (most cited first, 1 of 1)

- [Cultural Counterfactuals: Evaluating Cultural Biases in Large Vision-Language Models with Counterfactual Examples](https://mlsanity.com/paper/2603.02370/) (2026)

## Abstract (arXiv)

As large vision language models(LVLMs) rapidly advance, concerns about their potential to learn and generate social biases and stereotypes are increasing. Previous studies on LVLM's stereotypes face two primary limitations: metrics that overlooked the importance of content words, and datasets that overlooked the effect of color. To address these limitations, this study introduces new evaluation metrics based on the Stereotype Content Model (SCM). We also propose BASIC, a benchmark for assessing gender, race, and color stereotypes. Using SCM metrics and BASIC, we conduct a study with eight LVLMs to discover stereotypes. As a result, we found three findings. (1) The SCM-based evaluation is effective in capturing stereotypes. (2) LVLMs exhibit color stereotypes in the output along with gender and race ones. (3) Interaction between model architecture and parameter sizes seems to affect stereotypes. We release BASIC publicly on [anonymized for review].