mlsanityData to 14 Oct 2026

Traveling Across Languages: Benchmarking Cross-Lingual Consistency in Multimodal LLMs

arXiv 2505.15075 ↗First posted 21 May 2025cs.CL cs.AI cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The rapid evolution of multimodal large language models (MLLMs) has significantly enhanced their real-world applications. However, achieving consistent performance across languages, especially when integrating cultural knowledge, remains a significant challenge. To better assess this issue, we introduce two new benchmarks: KnowRecall and VisRecall, which evaluate cross-lingual consistency in MLLMs. KnowRecall is a visual question answering benchmark designed to measure factual knowledge consistency in 15 languages, focusing on cultural and historical questions about global landmarks. VisRecall assesses visual memory consistency by asking models to describe landmark appearances in 9 languages without access to images. Experimental results reveal that state-of-the-art MLLMs, including proprietary ones, still struggle to achieve cross-lingual consistency. This underscores the need for more robust approaches that produce truly multilingual and culturally aware models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 52 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

35 of 52 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model9 Jan
  2. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages25 Nov
  3. 2024Benchmarking Vision Language Models for Cultural Understanding15 Jul
  4. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark10 Jun

Other references · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2015VQA: Visual Question Answeringcited by 2,506
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  12. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
19 more
  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  3. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  4. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 309
  5. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  6. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  7. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  8. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 118
  9. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  10. 2024On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neuronscited by 83
  11. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  12. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 59
  13. 2024Should We Respect LLMs? A Cross-Lingual Study on the Influence of Prompt Politeness on LLM Performancecited by 41
  14. 2024jina-clip-v2: Multilingual Multimodal Embeddings for Text and Imagescited by 37
  15. 2025Aya Vision: Advancing the Frontier of Multilingual Multimodalitycited by 33
  16. 2024Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowlycited by 27
  17. 2024SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMscited by 12
  18. 2025CALM: Unleashing the Cross-Lingual Self-Aligning Ability of Language Model Question Answeringcited by 10
  19. 20241+1>2: Can Large Language Models Serve as Cross-Lingual Knowledge Aggregators?cited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questionscited by 4
  2. 2025Multilingual Vision-Language Models, A Surveycited by 1
  3. 2026SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Grounding Multilingual Multimodal LLMs With Cultural Knowledge
  2. 2025BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models
  3. 2024Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective
  4. 2026CrossCult-KIBench: A Benchmark for Cross-Cultural Knowledge Insertion in MLLMs
  5. 2025LinguaMark: Do Multimodal Models Speak Fairly? A Benchmark-Based Evaluation
  6. 2025MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation
  7. 2025M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAG
  8. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs