mlsanityData to 7 Oct 2026

Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learning

arXiv 2503.18484 ↗First posted 24 Mar 2025cs.CV cs.CLcited by 4

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Evaluating the multilingual capabilities of Large Vision-Language Models (LVLMs) remains challenging because most benchmarks rely on non-parallel corpora, making it unclear whether cross-lingual performance gaps reflect model limitations or dataset inconsistencies. To address this, we introduce PM4Bench, the first multimodal, multilingual, multi-task benchmark built on a strictly parallel 10-language corpus, enabling fair, apples-to-apples cross-lingual comparison of model performance. We further introduce a vision setting that embeds textual inputs directly into images, better approximating deployment scenarios where LVLM-driven agents interact with virtual or physical environments through unified visual observations. Experiments with 10 LVLMs reveal that OCR is a key factor behind cross-lingual disparity when textual content is rendered visually. Motivated by this, we design an OCR-centric GRPO training strategy using fully synthesized, label-free OCR data, without expensive task-specific VQA supervision. The resulting model improves general multilingual VQA capability, reduces cross-lingual disparities under the vision setting, and transfers gains beyond PM4Bench. This methodology offers an efficient, label-free pathway toward more equitable multilingual deployment of LVLM-driven agents.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 46 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

24 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages25 Nov
  2. 2024M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks4 Jul

Other references · 22

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  6. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  7. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  8. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  9. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  10. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  11. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  12. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
10 more
  1. 2024How do Large Language Models Handle Multilingualism?cited by 148
  2. 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
  3. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
  4. 2024CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacycited by 80
  5. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  6. 2024MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimizationcited by 68
  7. 2024Question Translation Training for Better Multilingual Reasoningcited by 41
  8. 2024P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMscited by 29
  9. 2024Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlationscited by 14
  10. 2024The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insightscited by 7

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
  2. 2025Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation
  3. 2025TowerVision: Understanding and Improving Multilinguality in Vision-Language Models
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  5. 2025VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages
  6. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  7. 2026Multilingual Training and Evaluation Resources for Vision-Language Models
  8. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models