# Breaking Language Barriers or Reinforcing Bias? A Study of Gender and Racial Disparities in Multilingual Contrastive Vision Language Models

arXiv: https://arxiv.org/abs/2505.14160 · HTML: https://mlsanity.com/paper/2505.14160/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Social bias and fairness in VLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 20 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Social bias and fairness in VLMs](https://mlsanity.com/mllm/social-bias-and-fairness-in-vlms/)

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [Debiasing Vision-Language Models via Biased Prompts](https://mlsanity.com/paper/2302.00070/) (2023)
- [Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset](https://mlsanity.com/paper/2205.12522/) (2022)
- [DeAR: Debiasing Vision-Language Models with Additive Residuals](https://mlsanity.com/paper/2303.10431/) (2023)
- [FairCLIP: Harnessing Fairness in Vision-Language Learning](https://mlsanity.com/paper/2403.19949/) (2024)
- [Comparing Biases and the Impact of Multilingual Training across Multiple Languages](https://mlsanity.com/paper/2305.11242/) (2023)
- [CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?](https://mlsanity.com/paper/2403.04547/) (2024)
- [MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs](https://mlsanity.com/paper/2406.07243/) (2024)
- [Identifying Implicit Social Biases in Vision-Language Models](https://mlsanity.com/paper/2411.00997/) (2024)
- [FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSs](https://mlsanity.com/paper/2403.15593/) (2024)
- [Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scale](https://mlsanity.com/paper/2305.13198/) (2023)
- [NLLB-CLIP -- train performant multilingual image retrieval model on a budget](https://mlsanity.com/paper/2309.01859/) (2023)
- [Do Multilingual Large Language Models Mitigate Stereotype Bias?](https://mlsanity.com/paper/2407.05740/) (2024)
- [Mitigating Social Bias in Large Language Models: A Multi-Objective Approach within a Multi-Agent Framework](https://mlsanity.com/paper/2412.15504/) (2024)
- [FairPIVARA: Reducing and Assessing Biases in CLIP-Based Multimodal Models](https://mlsanity.com/paper/2409.19474/) (2024)

## Built on by (most cited first, 2 of 2)

- [Investigating Social Bias in Narrative Image Generation](https://mlsanity.com/paper/2608.01780/) (2026)
- [GLEaN: A Text-to-image Bias Detection Approach for Public Comprehension](https://mlsanity.com/paper/2604.09923/) (2026)

## Abstract (arXiv)

Multilingual vision-language models (VLMs) promise universal image-text retrieval, yet their social biases remain underexplored. We perform the first systematic audit of four public multilingual CLIP variants: M-CLIP, NLLB-CLIP, CAPIVARA-CLIP, and the debiased SigLIP-2, covering ten languages that differ in resource availability and morphological gender marking. Using balanced subsets of FairFace and the PATA stereotype suite in a zero-shot setting, we quantify race and gender bias and measure stereotype amplification. Contrary to the intuition that multilinguality mitigates bias, every model exhibits stronger gender skew than its English-only baseline. CAPIVARA-CLIP shows its largest biases precisely in the low-resource languages it targets, while the shared encoder of NLLB-CLIP and SigLIP-2 transfers English gender stereotypes into gender-neutral languages; loosely coupled encoders largely avoid this leakage. Although SigLIP-2 reduces agency and communion skews, it inherits -- and in caption-sparse contexts (e.g., Xhosa) amplifies -- the English anchor's crime associations. Highly gendered languages consistently magnify all bias types, yet gender-neutral languages remain vulnerable whenever cross-lingual weight sharing imports foreign stereotypes. Aggregated metrics thus mask language-specific hot spots, underscoring the need for fine-grained, language-aware bias evaluation in future multilingual VLM research.