mlsanityData to 14 Oct 2026

LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation

arXiv 2507.07274 ↗First posted 9 Jul 2025cs.CV cs.AI cs.CLcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large Multimodal Models (LMMs) are typically trained on vast corpora of image-text data but are often limited in linguistic coverage, leading to biased and unfair outputs across languages. While prior work has explored multimodal evaluation, less emphasis has been placed on assessing multilingual capabilities. In this work, we introduce LinguaMark, a benchmark designed to evaluate state-of-the-art LMMs on a multilingual Visual Question Answering (VQA) task. Our dataset comprises 6,875 image-text pairs spanning 11 languages and five social attributes. We evaluate models using three key metrics: Bias, Answer Relevancy, and Faithfulness. Our findings reveal that closed-source models generally achieve the highest overall performance. Both closed-source (GPT-4o and Gemini2.5) and open-source models (Gemma3, Qwen2.5) perform competitively across social attributes, and Qwen2.5 demonstrates strong generalization across multiple languages. We release our benchmark and evaluation code to encourage reproducibility and further research.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 35 references that are in the map. First posted on arXiv 9 Jul 2025.

Builds on

30 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2023Multimodal Large Language Models: A Survey22 Nov
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug

Other references · 27

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2015VQA: Visual Question Answeringcited by 2,506
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  6. 2025Gemma 3 Technical Reportcited by 1,824
  7. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  8. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  9. 2024Phi-4 Technical Reportcited by 711
  10. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  11. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  12. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
15 more
  1. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  2. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  3. 2022MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languagescited by 146
  4. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
  5. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  6. 2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
  7. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languagescited by 55
  8. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  9. 2024EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Modelscited by 41
  10. 2025A Survey of Multilingual Reasoning in Language Modelscited by 33
  11. 2025BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Modelscited by 23
  12. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluationcited by 19
  13. 2024FAIR Enough: How Can We Develop and Assess a FAIR-Compliant Dataset for Large Language Models' Training?cited by 9
  14. 2024AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Modelscited by 8
  15. 2025MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matchingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026M3Kang: Evaluating Multilingual Multimodal Mathematical Reasoning in Vision-Language Modelscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAG
  2. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  3. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation
  4. 2025Grounding Multilingual Multimodal LLMs With Cultural Knowledge
  5. 2025UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation
  6. 2024A Survey on Benchmarks of Multimodal Large Language Models
  7. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark
  8. 2024M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models