mlsanityData to 7 Oct 2026

VisNumBench: Evaluating Number Sense of Multimodal Large Language Models

arXiv 2503.14939 ↗First posted 19 Mar 2025cs.CVcited by 12

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Can Multimodal Large Language Models (MLLMs) develop an intuitive number sense similar to humans? Targeting this problem, we introduce Visual Number Benchmark (VisNumBench) to evaluate the number sense abilities of MLLMs across a wide range of visual numerical tasks. VisNumBench consists of about 1,900 multiple-choice question-answer pairs derived from both synthetic and real-world visual data, covering seven visual numerical attributes and four types of visual numerical estimation tasks. Our experiments on VisNumBench led to the following key findings: (i) The 17 MLLMs we tested, including open-source models such as Qwen2.5-VL and InternVL2.5, as well as proprietary models like GPT-4o and Gemini 2.0 Flash, perform significantly below human levels in number sense-related tasks. (ii) Multimodal mathematical models and multimodal chain-of-thought (CoT) models did not exhibit significant improvements in number sense abilities. (iii) Stronger MLLMs with larger parameter sizes and broader general abilities demonstrate modest gains in number sense abilities. We believe VisNumBench will serve as a valuable resource for the research community, encouraging further advancements in enhancing MLLMs' number sense abilities. Code and dataset are available at https://wwwtttjjj.github.io/VisNumBench/.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 53 references that are in the map. First posted on arXiv 19 Mar 2025.

Builds on

40 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Step15 Nov
  2. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models25 Jun
  3. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?21 Mar
  4. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 34

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  11. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  12. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
22 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  4. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  5. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  6. 2020TartanAir: A Dataset to Push the Limits of Visual SLAMcited by 494
  7. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  8. 2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
  9. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  10. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  11. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  12. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  13. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoningcited by 119
  14. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  15. 2024MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Datacited by 51
  16. 2022OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regressioncited by 44
  17. 2024VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Informationcited by 39
  18. 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Modelscited by 32
  19. 2023Learning-to-Rank Meets Language: Boosting Language-Driven Ordering Alignment for Ordinal Classificationcited by 21
  20. 2024Teach CLIP to Develop a Number Sense for Ordinal Regressioncited by 20
  21. 2015An Image is Worth More than a Thousand Favorites: Surfacing the Hidden Beauty of Flickr Picturescited by 18
  22. 2024Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiadscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 7.

12 papers.

  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  2. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 95
  3. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  4. 2026MM-Zero: Self-Evolving Multi-Model Vision Language Models From Zero Datacited by 16
  5. 2026Active Zero: Self-Evolving Vision-Language Models through Active Environment Explorationcited by 7
  6. 2025LlamaSeg: Image Segmentation via Autoregressive Mask Generationcited by 4
  7. 2026OddGridBench: Exposing the Lack of Fine-Grained Visual Discrepancy Sensitivity in Multimodal Large Language Modelscited by 4
  8. 2026GPT4o-Receipt: A Dataset and Human Study for AI-Generated Document Forensicscited by 4
  9. 2026PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoningcited by 3
  10. 2026Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defensecited by 2
  11. 2025Grounding Multimodal Large Language Models with Quantitative Skin Attributes: A Retrieval Studycited by 0
  12. 2026Position: The Systemic Lack of Agency in Visual Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models
  2. 2025Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency
  3. 2025MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts
  4. 2024Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens
  5. 2025Large Language Models in Numberland: A Quick Test of Their Numerical Reasoning Abilities
  6. 2025MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Models
  7. 2025The Role of Visual Modality in Multimodal Mathematical Reasoning: Challenges and Insights
  8. 2026SenseMath: Do LLMs Have Number Sense? Evaluating Shortcut Use, Judgment, and Generation