mlsanityData to 14 Oct 2026

Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation

arXiv 2606.17188 ↗First posted 15 Jun 2026cs.CV cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Current multilingual evaluations for Vision-Language Models (VLMs) assume a one-to-one mapping between language and orthography, overlooking billions of users of multi-script languages. We introduce PuMVR (Punjabi Multimodal Visual Reasoning), a benchmark of 1,000 strictly parallel image-text instances across Punjabi's three active scripts: Gurmukhi, Shahmukhi, and Roman. Evaluating 10 state-of-the-art VLMs, we expose a substantial and systematic Script Gap. Models frequently solve visual tasks in one script while failing identical tasks in another, with accuracy deltas reaching 16%. Crucially, visual input boosts absolute performance uniformly yet does not close the orthographic gap. Furthermore, cross-script in-context transfer is highly brittle, exposing script-locked knowledge representation. Supported by McNemar tests across all script pairs, our findings demonstrate that current "multilingual" VLMs are not truly multi-script. We propose the Script Consistency Rate (SCR), which falls as low as 24.8% on our benchmark, as a mandatory metric for script-agnostic evaluation to ensure equitable AI access. Data and code are available at: https://github.com/prabhjotschugh/Not-Truly-Multilingual-PuMVR.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 20 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

15 of 20 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs6 Nov
  2. 2025BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models13 Oct
  3. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages25 Nov
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug

Other references · 11

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  3. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  4. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  5. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  6. 2021Broaden the Vision: Geo-Diverse Visual Commonsense Reasoningcited by 59
  7. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  8. 2024PALO: A Polyglot Large Multimodal Model for 5B Peoplecited by 34
  9. 2025MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matchingcited by 5
  10. 2025Script Gap: Evaluating LLM Triage on Indian Languages in Native vs Romanized Scripts in a Real World Settingcited by 5
  11. 2024CORI: CJKV Benchmark with Romanization Integration -- A step towards Cross-lingual Transfer Beyond Textual Scriptscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond 'One Language, One Script': Quantifying Orthographic Bias in Multilingual VLMs with PuMVR
  2. 2025VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages
  3. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  4. 2026Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
  5. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
  6. 2026UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking
  7. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models
  8. 2026Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models