mlsanityData to 7 Oct 2026

M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks

arXiv 2407.03791 ↗First posted 4 Jul 2024cs.CLcited by 27

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Since the release of ChatGPT, the field of Natural Language Processing has experienced rapid advancements, particularly in Large Language Models (LLMs) and their multimodal counterparts, Large Multimodal Models (LMMs). Despite their impressive capabilities, LLMs often exhibit significant performance disparities across different languages and cultural contexts, as demonstrated by various text-only benchmarks. However, current research lacks such benchmarks for multimodal visio-linguistic settings. This work fills this gap by introducing M5, the first comprehensive benchmark designed to evaluate LMMs on diverse vision-language tasks within a multilingual and multicultural context. M5 includes eight datasets covering five tasks and 41 languages, with a focus on underrepresented languages and culturally diverse images. Furthermore, we introduce two novel datasets, M5-VGR and M5-VLOD, including a new Visio-Linguistic Outlier Detection task, in which all evaluated open-source models fail to significantly surpass the random baseline. Through extensive evaluation and analyses, we highlight substantial task-agnostic performance disparities between high- and low-resource languages. Moreover, we show that larger models do not necessarily outperform smaller ones in a multilingual setting.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 52 references that are in the map. First posted on arXiv 4 Jul 2024.

Builds on

29 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  2. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs13 Jul

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  12. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
15 more
  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  2. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  3. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  4. 2024Yi: Open Foundation Models by 01.AIcited by 706
  5. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  6. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  7. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  8. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  9. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  10. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  11. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  12. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  13. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  14. 2024CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmarkcited by 54
  15. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 18; 2026: 6.

27 papers.

Sort:

By area:

  1. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  2. 2025MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMscited by 13
  3. 2025DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culturecited by 13
  4. 2025Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lenscited by 13
  5. 2025CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Interventioncited by 11
  6. 2025Pearl: A Multimodal Culturally-Aware Arabic Instruction Datasetcited by 6
  7. 2025Quantifying Language Disparities in Multilingual Large Language Modelscited by 6
  8. 2025MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matchingcited by 5
  9. 2025Hanfu-Bench: A Multimodal Benchmark on Cross-Temporal Cultural Understanding and Transcreationcited by 5
  10. 2024Benchmarking Multimodal Models for Ukrainian Language Understanding Across Academic and Cultural Domainscited by 4
  11. 2025Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learningcited by 4
  12. 2025Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularizationcited by 4
15 more
  1. 2025IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluationcited by 4
  2. 2025What are Foundation Models Cooking in the Post-Soviet World?cited by 3
  3. 2025Language-Specific Layer Matters: Efficient Multilingual Enhancement for Large Vision-Language Modelscited by 3
  4. 2025PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Modelscited by 3
  5. 2026Afri-MCQA: Multimodal Cultural Question Answering for African Languagescited by 2
  6. 2026M3Kang: Evaluating Multilingual Multimodal Mathematical Reasoning in Vision-Language Modelscited by 2
  7. 2025Investigating the Effect of Parallel Data in the Cross-Lingual Transfer for Vision-Language Encoderscited by 1
  8. 2025Multilingual Vision-Language Models, A Surveycited by 1
  9. 2025ELR-1000: A Community-Generated Dataset for Endangered Indic Indigenous Languagescited by 1
  10. 2026DiMtextsuperscript3: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Mergingcited by 1
  11. 2024Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspectivecited by 0
  12. 2025Large Multi-modal Model Cartographic Map Comprehension for Textual Locality Georeferencingcited by 0
  13. 2026PUMA: A Polish Benchmark for Culturally Grounded Multimodal Understandingcited by 0
  14. 2026PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoningcited by 0
  15. 2026CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  2. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  4. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
  5. 2026FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs
  6. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Models
  7. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models
  8. 2025MMLongCite: A Benchmark for Evaluating Fidelity of Long-Context Vision-Language Models