mlsanityData to 7 Oct 2026

What Is Missing in Multilingual Visual Reasoning and How to Fix It

arXiv 2403.01404 ↗First posted 3 Mar 2024cs.CLcited by 7

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

NLP models today strive for supporting multiple languages and modalities, improving accessibility for diverse users. In this paper, we evaluate their multilingual, multimodal capabilities by testing on a visual reasoning task. We observe that proprietary systems like GPT-4V obtain the best performance on this task now, but open models lag in comparison. Surprisingly, GPT-4V exhibits similar performance between English and other languages, indicating the potential for equitable system development across languages. Our analysis on model failures reveals three key aspects that make this task challenging: multilinguality, complex reasoning, and multimodality. To address these challenges, we propose three targeted interventions including a translate-test approach to tackle multilinguality, a visual programming approach to break down complex reasoning, and a method that leverages image captioning to address multimodality. Our interventions achieve the best open performance on this task in a zero-shot setting, boosting open models LLaVA-v1.5-13B by 13.4%, LLaVA-v1.6-34B by 20.3%, and Qwen-VL by 16.7%, while also minorly improving GPT-4V's performance.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 53 references that are in the map. First posted on arXiv 3 Mar 2024.

Builds on

19 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models8 Mar

Other references · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  10. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  11. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  12. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
6 more
  1. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  2. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  3. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  4. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  5. 2023Benchmarking Machine Translation with Cultural Awarenesscited by 53
  6. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 1.

7 papers.

  1. 2025A Survey of Multilingual Reasoning in Language Modelscited by 31
  2. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understandingcited by 17
  3. 2024Large Language Models Are Cross-Lingual Knowledge-Free Reasonerscited by 16
  4. 2024Why do LLaVA Vision-Language Models Reply to Images in English?cited by 11
  5. 2025Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularizationcited by 4
  6. 2025Multilingual Vision-Language Models, A Surveycited by 1
  7. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models
  2. 2024Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models
  3. 2023Lost in Translation: When GPT-4V(ision) Can't See Eye to Eye with Text. A Vision-Language-Consistency Analysis of VLLMs and Beyond
  4. 2026MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation
  5. 2025VisualQuest: A Benchmark for Abstract Visual Reasoning in MLLMs
  6. 2025VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark
  7. 2026Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation
  8. 2025Rethinking Multilingual Vision-Language Translation: Dataset, Evaluation, and Adaptation