mlsanityData to 7 Oct 2026

MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching

arXiv 2502.12852 ↗First posted 18 Feb 2025cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Existing multilingual vision-language (VL) benchmarks often only cover a handful of languages. Consequently, evaluations of large vision-language models (LVLMs) predominantly target high-resource languages, underscoring the need for evaluation data for low-resource languages. To address this limitation, we introduce MVL-SIB, a massively multilingual vision-language benchmark that evaluates both cross-modal and text-only topical matching across 205 languages -- over 100 more than the most multilingual existing VL benchmarks encompass. We then benchmark a range of of open-weight LVLMs together with GPT-4o(-mini) on MVL-SIB. Our results reveal that LVLMs struggle in cross-modal topic matching in lower-resource languages, performing no better than chance on languages like N'Koo. Our analysis further reveals that VL support in LVLMs declines disproportionately relative to textual support for lower-resource languages, as evidenced by comparison of cross-modal and text-only topical matching performance. We further observe that open-weight LVLMs do not benefit from representing a topic with more than one image, suggesting that these models are not yet fully effective at handling multi-image tasks. By correlating performance on MVL-SIB with other multilingual VL benchmarks, we highlight that MVL-SIB serves as a comprehensive probe of multilingual VL understanding in LVLMs.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 22 of its 33 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

22 of 33 references.

References · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  10. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  11. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  12. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
10 more
  1. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  2. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  3. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  4. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  5. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 63
  6. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  7. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41
  8. 2024African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classificationcited by 31
  9. 2024M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Taskscited by 27
  10. 2017Baselines and test data for cross-lingual inferencecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluationcited by 1
  2. 2025Multilingual Vision-Language Models, A Surveycited by 1
  3. 2026Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluationcited by 0
  4. 2026Beyond 'One Language, One Script': Quantifying Orthographic Bias in Multilingual VLMs with PuMVRcited by 0
  5. 2026Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQAcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  2. 2025Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation
  3. 2025Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learning
  4. 2025uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data
  5. 2024Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models
  6. 2024EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models
  7. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
  8. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models