# Evaluating Perspectival Biases in Cross-Modal Retrieval

arXiv: https://arxiv.org/abs/2510.26861 · HTML: https://mlsanity.com/paper/2510.26861/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 30 Oct 2025.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [ColPali: Efficient Document Retrieval with Vision Language Models](https://mlsanity.com/paper/2407.01449/) (2024)
- [Visually Grounded Reasoning across Languages and Cultures](https://mlsanity.com/paper/2109.13238/) (2021)
- [CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark](https://mlsanity.com/paper/2406.05967/) (2024)
- [Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese](https://mlsanity.com/paper/2211.01335/) (2022)
- [Benchmarking Vision Language Models for Cultural Understanding](https://mlsanity.com/paper/2407.10920/) (2024)
- [Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset](https://mlsanity.com/paper/2205.12522/) (2022)
- [jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval](https://mlsanity.com/paper/2506.18902/) (2025)
- [Assessing Multilingual Fairness in Pre-trained Multimodal Representations](https://mlsanity.com/paper/2106.06683/) (2021)
- [Fairness and Bias in Multimodal AI: A Survey](https://mlsanity.com/paper/2406.19097/) (2024)
- [MIRACL-VISION: A Large, multilingual, visual document retrieval benchmark](https://mlsanity.com/paper/2505.11651/) (2025)
- [RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding](https://mlsanity.com/paper/2505.14462/) (2025)
- [FooDI-ML: a large multi-language dataset of food, drinks and groceries images and descriptions](https://mlsanity.com/paper/2110.02035/) (2021)
- [Mitigating Language Bias in Cross-Lingual Job Retrieval: A Recruitment Platform Perspective](https://mlsanity.com/paper/2502.03220/) (2025)

## Abstract (arXiv)

Multimodal retrieval systems are expected to operate in a semantic space, agnostic to the language or cultural origin of the query. In practice, however, retrieval outcomes systematically reflect perspectival biases: deviations shaped by linguistic prevalence and cultural associations. We introduce the Cross-Cultural, Cross-Modal, Cross-lingual Multimodal (3XCM) benchmark to isolate these effects. Results from our studies indicate that, for image-to-text retrieval, models tend to favor entries from prevalent languages over those that are semantically faithful. For text-to-image retrieval, we observe a consistent "tugging effect" in the joint embedding space between semantic alignment and language-conditioned cultural association. When semantic representations are insufficiently resolved, particularly in low-resource languages, similarity is increasingly governed by culturally familiar visual patterns, leading to systematic association bias in retrieval. Our findings suggest that achieving equitable multimodal retrieval necessitates targeted strategies that explicitly decouple language from culture, rather than relying solely on broader data exposure. This work highlights the need to treat linguistic and cultural biases as distinct, measurable challenges in multimodal representation learning.