# MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching

arXiv: https://arxiv.org/abs/2502.12852 · HTML: https://mlsanity.com/paper/2502.12852/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Visual Entailment: A Novel Task for Fine-Grained Image Understanding](https://mlsanity.com/paper/1901.06706/) (2019)
- [Visually Grounded Reasoning across Languages and Cultures](https://mlsanity.com/paper/2109.13238/) (2021)
- [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://mlsanity.com/paper/2309.04662/) (2023)
- [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://mlsanity.com/paper/2309.07445/) (2023)
- [Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset](https://mlsanity.com/paper/2205.12522/) (2022)
- [xGQA: Cross-Lingual Visual Question Answering](https://mlsanity.com/paper/2109.06082/) (2021)
- [Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages](https://mlsanity.com/paper/2410.16153/) (2024)
- [IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages](https://mlsanity.com/paper/2201.11732/) (2022)
- [mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs](https://mlsanity.com/paper/2307.06930/) (2023)
- [African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification](https://mlsanity.com/paper/2406.14496/) (2024)
- [M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks](https://mlsanity.com/paper/2407.03791/) (2024)
- [Baselines and test data for cross-lingual inference](https://mlsanity.com/paper/1704.05347/) (2017)

## Built on by (most cited first, 5 of 5)

- [LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation](https://mlsanity.com/paper/2507.07274/) (2025)
- [Multilingual Vision-Language Models, A Survey](https://mlsanity.com/paper/2509.22123/) (2025)
- [Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation](https://mlsanity.com/paper/2606.17188/) (2026)
- [Beyond 'One Language, One Script': Quantifying Orthographic Bias in Multilingual VLMs with PuMVR](https://mlsanity.com/paper/2606.20770/) (2026)
- [Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQA](https://mlsanity.com/paper/2608.22363/) (2026)

## Abstract (arXiv)

Existing multilingual vision-language (VL) benchmarks often only cover a handful of languages. Consequently, evaluations of large vision-language models (LVLMs) predominantly target high-resource languages, underscoring the need for evaluation data for low-resource languages. To address this limitation, we introduce MVL-SIB, a massively multilingual vision-language benchmark that evaluates both cross-modal and text-only topical matching across 205 languages -- over 100 more than the most multilingual existing VL benchmarks encompass. We then benchmark a range of of open-weight LVLMs together with GPT-4o(-mini) on MVL-SIB. Our results reveal that LVLMs struggle in cross-modal topic matching in lower-resource languages, performing no better than chance on languages like N'Koo. Our analysis further reveals that VL support in LVLMs declines disproportionately relative to textual support for lower-resource languages, as evidenced by comparison of cross-modal and text-only topical matching performance. We further observe that open-weight LVLMs do not benefit from representing a topic with more than one image, suggesting that these models are not yet fully effective at handling multi-image tasks. By correlating performance on MVL-SIB with other multilingual VL benchmarks, we highlight that MVL-SIB serves as a comprehensive probe of multilingual VL understanding in LVLMs.