Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learning
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Evaluating the multilingual capabilities of Large Vision-Language Models (LVLMs) remains challenging because most benchmarks rely on non-parallel corpora, making it unclear whether cross-lingual performance gaps reflect model limitations or dataset inconsistencies. To address this, we introduce PM4Bench, the first multimodal, multilingual, multi-task benchmark built on a strictly parallel 10-language corpus, enabling fair, apples-to-apples cross-lingual comparison of model performance. We further introduce a vision setting that embeds textual inputs directly into images, better approximating deployment scenarios where LVLM-driven agents interact with virtual or physical environments through unified visual observations. Experiments with 10 LVLMs reveal that OCR is a key factor behind cross-lingual disparity when textual content is rendered visually. Motivated by this, we design an OCR-centric GRPO training strategy using fully synthesized, label-free OCR data, without expensive task-specific VQA supervision. The resulting model improves general multilingual VQA capability, reduces cross-lingual disparities under the vision setting, and transfers gains beyond PM4Bench. This methodology offers an efficient, label-free pathway toward more equitable multilingual deployment of LVLM-driven agents.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 46 references that are in the map. First posted on arXiv 24 Mar 2025.
Builds on
24 of 46 references.
Closest ancestors · 2
Same direction, newest first
- 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages25 Nov
- 2024M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks4 Jul
Other references · 22
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
- 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
- 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
- 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
- 2021Visually Grounded Reasoning across Languages and Culturescited by 188
- 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
10 more
- 2024How do Large Language Models Handle Multilingualism?cited by 148
- 2024Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Modelscited by 124
- 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 89
- 2024CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacycited by 80
- 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
- 2024MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimizationcited by 68
- 2024Question Translation Training for Better Multilingual Reasoningcited by 41
- 2024P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMscited by 29
- 2024Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlationscited by 14
- 2024The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insightscited by 7
Built on by
Citing papers per year: 2025: 4.
4 papers.
- 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
- 2025GTR-CoT: Graph Traversal as Visual Chain of Thought for Molecular Structure Recognitioncited by 7
- 2025RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioningcited by 3
- 2025MSA at ImageCLEF 2025 Multimodal Reasoning: Multilingual Multimodal Reasoning With Ensemble Vision Language Modelscited by 1
- 2025RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioning4 Nov 2025
- 2025MSA at ImageCLEF 2025 Multimodal Reasoning: Multilingual Multimodal Reasoning With Ensemble Vision Language Models15 Jul 2025
- 2025GTR-CoT: Graph Traversal as Visual Chain of Thought for Molecular Structure Recognition9 Jun 2025
- 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models8 May 2025
Similar, not cited
Close in wording, no citation either way
- 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
- 2025Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation
- 2025TowerVision: Understanding and Improving Multilinguality in Vision-Language Models
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
- 2025VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages
- 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
- 2026Multilingual Training and Evaluation Resources for Vision-Language Models
- 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models