mlsanityData to 7 Oct 2026

MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?

arXiv 2403.14624 ↗First posted 21 Mar 2024cs.CV cs.AI cs.CLcited by 573

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The remarkable progress of Multi-modal Large Language Models (MLLMs) has garnered unparalleled attention, due to their superior performance in visual contexts. However, their capabilities in visual math problem-solving remain insufficiently evaluated and understood. We investigate current benchmarks to incorporate excessive visual content within textual questions, which potentially assist MLLMs in deducing answers without truly interpreting the input diagrams. To this end, we introduce MathVerse, an all-around visual math benchmark designed for an equitable and in-depth evaluation of MLLMs. We meticulously collect 2,612 high-quality, multi-subject math problems with diagrams from publicly available sources. Each problem is then transformed by human annotators into six distinct versions, each offering varying degrees of information content in multi-modality, contributing to 15K test samples in total. This approach allows MathVerse to comprehensively assess whether and how much MLLMs can truly understand the visual diagrams for mathematical reasoning. In addition, we propose a Chain-of-Thought (CoT) evaluation strategy for a fine-grained assessment of the output answers. Rather than naively judging True or False, we employ GPT-4(V) to adaptively extract crucial reasoning steps, and then score each step with detailed error analysis, which can reveal the intermediate CoT reasoning quality by MLLMs. We hope the MathVerse benchmark may provide unique insights to guide the future development of MLLMs. Project page: https://mathverse-cuhk.github.io

Key facts. Cited by 573 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 68 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

54 of 68 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise19 Dec
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2023Segment Anythingcited by 8,158
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  12. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
38 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2024Mixtral of Expertscited by 1,785
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  12. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  13. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  14. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  15. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  16. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  17. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  18. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  19. 2016Solving General Arithmetic Word Problemscited by 353
  20. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  21. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  22. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  23. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  24. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  25. 2023Personalize Segment Anything Model with One Shotcited by 250
  26. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
  27. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  28. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  29. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  30. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  31. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  32. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 155
  33. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  34. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
  35. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  36. 2022UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expressioncited by 137
  37. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  38. 2023VideoLLM: Modeling Video Sequence with Large Language Modelscited by 116

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 96; 2025: 258; 2026: 219.

573 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2025Qwen3-VL Technical Reportcited by 2,534
  3. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  6. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 653
  7. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  8. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  9. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  10. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  11. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
  12. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
38 more
  1. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 364
  2. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  3. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  4. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  5. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  6. 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 294
  7. 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?cited by 281
  8. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  9. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 277
  10. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  11. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  12. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  13. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  14. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  15. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 176
  16. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  17. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  18. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  19. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  20. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  21. 2025Thyme: Think Beyond Imagescited by 160
  22. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  23. 2025SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvementcited by 144
  24. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  25. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  26. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  27. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Modelscited by 133
  28. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  29. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  30. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  31. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  32. 2024AudioBench: A Universal Benchmark for Audio Large Language Modelscited by 124
  33. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  34. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  35. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
  36. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 111
  37. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 111
  38. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110

50 most cited of 573 All 573 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Math-Vision Diagrams: A Comprehensive Benchmark for Evaluating LLM Mathematical Diagram Generation Capabilities
  2. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
  4. 2025MathCanvas: Intrinsic Visual Chain-of-Thought for Multimodal Mathematical Reasoning
  5. 2025MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Models
  6. 2025The Role of Visual Modality in Multimodal Mathematical Reasoning: Challenges and Insights
  7. 2024Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark
  8. 2025MathSight: A Benchmark Exploring Have Vision-Language Models Really Seen in University-Level Mathematical Reasoning?