mlsanityData to 14 Oct 2026

VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models

arXiv 2504.15279 ↗First posted 21 Apr 2025cs.CVcited by 107

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Visual reasoning is a core component of human intelligence and a critical capability for advanced multimodal models. Yet current reasoning evaluations of multimodal large language models (MLLMs) often rely on text descriptions and allow language-based reasoning shortcuts, failing to measure genuine vision-centric reasoning. To address this, we introduce VisuLogic: a benchmark of 1,000 human-verified problems across six categories (e.g., quantitative shifts, spatial relations, attribute comparisons). These various types of questions can be evaluated to assess the visual reasoning capabilities of MLLMs from multiple perspectives. We evaluate leading MLLMs on this benchmark and analyze their results to identify common failure modes. Most models score below 30% accuracy-only slightly above the 25% random baseline and far below the 51.4% achieved by humans-revealing significant gaps in visual reasoning. Furthermore, we provide a supplementary training dataset and a reinforcement-learning baseline to support further progress.

Key facts. Cited by 107 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 69 of its 83 references that are in the map. First posted on arXiv 21 Apr 2025.

Builds on

69 of 83 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model10 Apr
  2. 2025OThink-MR1: Stimulating multimodal generalized reasoning capabilities via dynamic reinforcement learning20 Mar
  3. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization13 Mar
  4. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoning13 Mar
  5. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL10 Mar
  6. 2025Visual-RFT: Visual Reinforcement Fine-Tuning3 Mar
  7. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark9 Jan
  8. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  9. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts6 Jul
  10. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?1 Jul
  11. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  12. 2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models27 Nov
3 more
  1. 2023InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models20 Nov
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 54

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2025Qwen2.5-VL Technical Reportcited by 5,587
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2024GPT-4o System Cardcited by 4,588
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
38 more
  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  3. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  4. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  8. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  9. 2016Modeling Context in Referring Expressionscited by 1,063
  10. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  13. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  14. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  15. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  16. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  17. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  18. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 226
  19. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  20. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  21. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  22. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  23. 2023Structured Chain-of-Thought Prompting for Code Generationcited by 138
  24. 2023Self-planning Code Generation with Large Language Modelscited by 124
  25. 2023PointGPT: Auto-regressively Generative Pre-training from Point Cloudscited by 115
  26. 2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 113
  27. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 102
  28. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  29. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  30. 2023AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?cited by 81
  31. 2024Benchmarking and Improving Detail Image Captioncited by 81
  32. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  33. 2024ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipelinecited by 47
  34. 2023Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Modelscited by 39
  35. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarioscited by 35
  36. 2019Reflective Decoding Network for Image Captioningcited by 31
  37. 2023LogiCoT: Logical Chain-of-Thought Instruction-Tuningcited by 31
  38. 2023CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generationcited by 29

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 41; 2026: 66.

107 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,793
  2. 2025Thyme: Think Beyond Imagescited by 167
  3. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  4. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 84
  5. 2025Kwai Keye-VL 1.5 Technical Reportcited by 73
  6. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMscited by 59
  7. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 56
  8. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 49
  9. 2025Kwai Keye-VL Technical Reportcited by 47
  10. 2025Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmarkcited by 43
  11. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 36
  12. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 35
38 more
  1. 2026STEP3-VL-10B Technical Reportcited by 32
  2. 2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 30
  3. 2026MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methodscited by 30
  4. 2025V-Thinker: Interactive Thinking with Imagescited by 26
  5. 2025OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoningcited by 25
  6. 2025Simple o3: Towards Interleaved Vision-Language Reasoningcited by 24
  7. 2025ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Mapscited by 23
  8. 2025SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligencecited by 23
  9. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmarkcited by 21
  10. 2025RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learningcited by 21
  11. 2025When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thoughtcited by 21
  12. 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Imagecited by 20
  13. 2025R-4B: Incentivizing General-Purpose Auto-Thinking Capability in MLLMs via Bi-Mode Annealing and Reinforce Learningcited by 18
  14. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 18
  15. 2025Skywork-R1V3 Technical Reportcited by 16
  16. 2025Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Modelscited by 14
  17. 2025VisualSphinx: Large-Scale Synthetic Vision Logic Puzzles for RLcited by 12
  18. 2026Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decodingcited by 12
  19. 2026ERNIE 5.0 Technical Reportcited by 11
  20. 2025Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Modelscited by 10
  21. 2026MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Modelscited by 10
  22. 2026ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratchcited by 10
  23. 2026Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Modelscited by 10
  24. 2025Reinforcement Learning for Large Model: A Surveycited by 9
  25. 2025GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contextscited by 8
  26. 2025Revisiting the Necessity of Lengthy Chain-of-Thought in Vision-centric Reasoning Generalizationcited by 8
  27. 2026VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?cited by 8
  28. 2025VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoningcited by 7
  29. 2025MuSLR: Multimodal Symbolic Logical Reasoningcited by 7
  30. 2026MentisOculi: Revealing the Limits of Reasoning with Mental Imagerycited by 7
  31. 2026What's Holding Back Latent Visual Reasoning?cited by 7
  32. 2025GeoSketch: A Neural-Symbolic Approach to Geometric Multimodal Reasoning with Auxiliary Line Construction and Affine Transformationcited by 6
  33. 2026Learning Modal-Mixed Chain-of-Thought Reasoning with Latent Embeddingscited by 6
  34. 2026Beyond Mode Collapse: Distribution Matching for Diverse Reasoningcited by 6
  35. 2026Leveraging Latent Visual Reasoning in Silencecited by 5
  36. 2026MOSS-VL Technical Reportcited by 5
  37. 2025Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Samplingcited by 4
  38. 2026Milestones over Outcome: Unlocking Geometric Reasoning with Sub-Goal Verifiable Rewardcited by 4

50 most cited of 107 All 107 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning
  2. 2025Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward
  3. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models
  4. 2023Benchmarking Sequential Visual Input Reasoning and Prediction in Multimodal Large Language Models
  5. 2025VERIFY: A Benchmark of Visual Explanation and Reasoning for Investigating Multimodal Reasoning Fidelity
  6. 2026ViSTR-Bench: Can MLLMs Reason from Continuous Visual Cues in Dynamic Scenes?
  7. 2026Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
  8. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models