mlsanityData to 7 Oct 2026

Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset

arXiv 2402.14804 ↗First posted 22 Feb 2024cs.CV cs.AI cs.CLcited by 516

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Recent advancements in Large Multimodal Models (LMMs) have shown promising results in mathematical reasoning within visual contexts, with models approaching human-level performance on existing benchmarks such as MathVista. However, we observe significant limitations in the diversity of questions and breadth of subjects covered by these benchmarks. To address this issue, we present the MATH-Vision (MATH-V) dataset, a meticulously curated collection of 3,040 high-quality mathematical problems with visual contexts sourced from real math competitions. Spanning 16 distinct mathematical disciplines and graded across 5 levels of difficulty, our dataset provides a comprehensive and diverse set of challenges for evaluating the mathematical reasoning abilities of LMMs. Through extensive experimentation, we unveil a notable performance gap between current LMMs and human performance on MATH-V, underscoring the imperative for further advancements in LMMs. Moreover, our detailed categorization allows for a thorough error analysis of LMMs, offering valuable insights to guide future research and development. The project is available at https://mathvision-cuhk.github.io

Key facts. Cited by 516 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 46 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

22 of 46 references.

References · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  10. 2023PaLM 2 Technical Reportcited by 1,154
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  12. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
10 more
  1. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  2. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  3. 2023The Falcon Series of Open Language Modelscited by 526
  4. 2023Llemma: An Open Language Model For Mathematicscited by 385
  5. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  6. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  7. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  8. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  9. 2023NExT-Chat: An LMM for Chat, Detection and Segmentationcited by 79
  10. 2023Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Modelscited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 57; 2025: 244; 2026: 215.

516 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2025Qwen3-VL Technical Reportcited by 2,534
  3. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  5. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  6. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  7. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  8. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  9. 2025Qwen2.5-Omni Technical Reportcited by 704
  10. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  11. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  12. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
38 more
  1. 2025Qwen3-Omni Technical Reportcited by 451
  2. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  3. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  4. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  5. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  6. 2025Kimi-VL Technical Reportcited by 381
  7. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  8. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  9. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  10. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  11. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  12. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  13. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  14. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  15. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  16. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 171
  17. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  18. 2025Thyme: Think Beyond Imagescited by 160
  19. 2026Qwen3.5-Omni Technical Reportcited by 146
  20. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 144
  21. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  22. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  23. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  24. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  25. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Modelscited by 133
  26. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  27. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  28. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
  29. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 111
  30. 2025MiMo-VL Technical Reportcited by 106
  31. 2025Ovis2.5 Technical Reportcited by 103
  32. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 102
  33. 2026Kimi K3: Open Frontier Intelligencecited by 99
  34. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 92
  35. 2025Soft Adaptive Policy Optimizationcited by 89
  36. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  37. 2025Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learningcited by 86
  38. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 84

50 most cited of 516 All 516 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?
  2. 2025Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
  4. 2026M3Kang: Evaluating Multilingual Multimodal Mathematical Reasoning in Vision-Language Models
  5. 2026A Survey of Multimodal Mathematical Reasoning: From Perception, Alignment to Reasoning
  6. 2025MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Models
  7. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
  8. 2025Evaluating Visual Mathematics in Multimodal LLMs: A Multilingual Benchmark Based on the Kangaroo Tests