mlsanityData to 14 Oct 2026

GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoning

arXiv 2312.12241 ↗First posted 19 Dec 2023cs.CV cs.CLcited by 108

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Large language models have shown impressive results for multi-hop mathematical reasoning when the input question is only textual. Many mathematical reasoning problems, however, contain both text and image. With the ever-increasing adoption of vision language models (VLMs), understanding their reasoning abilities for such problems is crucial. In this paper, we evaluate the reasoning capabilities of VLMs along various axes through the lens of geometry problems. We procedurally create a synthetic dataset of geometry questions with controllable difficulty levels along multiple axes, thus enabling a systematic evaluation. The empirical results obtained using our benchmark for state-of-the-art VLMs indicate that these models are not as capable in subjects like geometry (and, by generalization, other topics requiring similar reasoning) as suggested by previous benchmarks. This is made especially clear by the construction of our benchmark at various depth levels, since solving higher-depth problems requires long chains of reasoning rather than additional memorized knowledge. We release the dataset for further research in this area.

Key facts. Cited by 108 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 47 of its 64 references that are in the map. First posted on arXiv 19 Dec 2023.

Builds on

47 of 64 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023A Multi-Modal Neural Geometric Solver with Textual Clauses Parsed from Diagram22 Feb
  2. 2022UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression6 Dec

Other references · 45

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  11. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
33 more
  1. 2023PaLM 2 Technical Reportcited by 1,162
  2. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  3. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  4. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  5. 2022Large Language Models Can Self-Improvecited by 664
  6. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  7. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  8. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  9. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 391
  10. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  11. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 334
  12. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 324
  13. 2019PlotQA: Reasoning over Scientific Plotscited by 321
  14. 2022Language Models are Realistic Tabular Data Generatorscited by 294
  15. 2022Teaching Small Language Models to Reasoncited by 284
  16. 2020Transformers as Soft Reasoners over Languagecited by 270
  17. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  18. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 228
  19. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 210
  20. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 176
  21. 2021Explaining Answers with Entailment Treescited by 152
  22. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  23. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoningcited by 120
  24. 2023Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examplescited by 84
  25. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 83
  26. 2022LAMBADA: Backward Chaining for Automated Reasoning in Natural Languagecited by 66
  27. 2021AR-LSAT: Investigating Analytical Reasoning of Textcited by 51
  28. 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Informationcited by 44
  29. 2020Critical Thinking for Language Modelscited by 31
  30. 2022GPT-3-driven pedagogical agents for training children's curious question-asking skillscited by 29
  31. 2022Penguins Don't Fly: Reasoning about Generics through Instantiations and Exceptionscited by 17
  32. 2022Neural Story Planningcited by 5
  33. 2023Evaluating the Capabilities of Multi-modal Reasoning Models with Synthetic Task Datacited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 37; 2025: 47; 2026: 24.

108 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  3. 2025Gemma 3 Technical Reportcited by 1,824
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  5. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  6. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  7. 2024What matters when building vision-language models?cited by 326
  8. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  9. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  10. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  11. 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 175
  12. 2024Building and better understanding vision-language models: insights and future directionscited by 173
38 more
  1. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 139
  2. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 127
  3. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
  4. 2025BIG-Bench Extra Hardcited by 125
  5. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 113
  6. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 113
  7. 2025Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Modelscited by 99
  8. 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 88
  9. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  10. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 66
  11. 2024A Survey on Deep Learning for Theorem Provingcited by 57
  12. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  13. 2025FineVision: Open Data Is All You Needcited by 53
  14. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  15. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  16. 2024GeoEval: Benchmark for Evaluating LLMs and Multi-Modal Models on Geometry Problem-Solvingcited by 45
  17. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  18. 2025VisuoThink: Empowering LVLM Reasoning with Multimodal Tree Searchcited by 36
  19. 2024Theorem-Validated Reverse Chain-of-Thought Problem Generation for Geometric Reasoningcited by 32
  20. 2024ReMI: A Dataset for Reasoning with Multiple Imagescited by 30
  21. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  22. 2024AutoGeo: Automating Geometric Image Dataset Creation for Enhanced Geometry Understandingcited by 29
  23. 2024Autoformalizing Euclidean Geometrycited by 27
  24. 2025GeoSense: Evaluating Identification and Application of Geometric Principles in Multimodal Reasoningcited by 25
  25. 2024MathGLM-Vision: Solving Mathematical Problems with Multi-Modal Large Language Modelcited by 21
  26. 2024ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Modelscited by 21
  27. 2024The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Modelscited by 17
  28. 2024Whiteboard-of-Thought: Thinking Step-by-Step Across Modalitiescited by 16
  29. 2025Open Eyes, Then Reason: Fine-grained Visual Mathematical Understanding in MLLMscited by 16
  30. 2025Athena: Enhancing Multimodal Reasoning with Data-efficient Process Reward Modelscited by 16
  31. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  32. 2024Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptionscited by 13
  33. 2025Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learnedcited by 13
  34. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 12
  35. 2024OmChat: A Recipe to Train Multimodal Language Models with Strong Long Context and Video Understandingcited by 11
  36. 2024Chimera: Improving Generalist Model with Domain-Specific Expertscited by 11
  37. 2025Corvid: Improving Multimodal Large Language Models Towards Chain-of-Thought Reasoningcited by 11
  38. 2026Vero: An Open RL Recipe for General Visual Reasoningcited by 11

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025GeoThought: A Dataset for Enhancing Mathematical Geometry Reasoning in Vision-Language Models
  2. 2025DynaSolidGeo: A Dynamic Benchmark for Genuine Spatial Mathematical Reasoning of VLMs in Solid Geometry
  3. 2024Beyond Lines and Circles: Unveiling the Geometric Reasoning Gap in Large Language Models
  4. 2026GeoReader: Learning Latent Geometry for Spatial Reasoning in Vision-Language Models
  5. 2024Improving Multimodal LLMs Ability In Geometry Problem Solving, Reasoning, And Multistep Scoring
  6. 2024GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-training
  7. 2026Make Geometry Matter for Spatial Reasoning
  8. 2025GanitBench: A bi-lingual benchmark for evaluating mathematical reasoning in Vision Language Models