mlsanityData to 14 Oct 2026

GeoEval: Benchmark for Evaluating LLMs and Multi-Modal Models on Geometry Problem-Solving

arXiv 2402.10104 ↗First posted 15 Feb 2024cs.AI cs.CLcited by 45

Not on the map. Why?

Cited by 28 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advancements in large language models (LLMs) and multi-modal models (MMs) have demonstrated their remarkable capabilities in problem-solving. Yet, their proficiency in tackling geometry math problems, which necessitates an integrated understanding of both textual and visual information, has not been thoroughly evaluated. To address this gap, we introduce the GeoEval benchmark, a comprehensive collection that includes a main subset of 2,000 problems, a 750 problems subset focusing on backward reasoning, an augmented subset of 2,000 problems, and a hard subset of 300 problems. This benchmark facilitates a deeper investigation into the performance of LLMs and MMs in solving geometry math problems. Our evaluation of ten LLMs and MMs across these varied subsets reveals that the WizardMath model excels, achieving a 55.67% accuracy rate on the main subset but only a 6.00% accuracy on the hard subset. This highlights the critical need for testing models against datasets on which they have not been pre-trained. Additionally, our findings indicate that GPT-series models perform more effectively on problems they have rephrased, suggesting a promising method for enhancing model capabilities.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 21 of its 40 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

21 of 40 references.

References · 21

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  7. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  8. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  9. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  10. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  11. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  12. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
9 more
  1. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 186
  2. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  3. 2022UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expressioncited by 137
  4. 2023GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoningcited by 108
  5. 2023A Multi-Modal Neural Geometric Solver with Textual Clauses Parsed from Diagramcited by 56
  6. 2023Forward-Backward Reasoning in Large Language Models for Mathematical Verificationcited by 40
  7. 2023LANS: A Layout-Aware Neural Solver for Plane Geometry Problemcited by 24
  8. 2021Sequence to General Tree: Knowledge-Guided Geometry Word Problem Solvingcited by 11
  9. 2024GAPS: Geometry-Aware Problem Solvercited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 24; 2026: 10.

45 papers.

Sort:

By area:

  1. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 98
  2. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 63
  3. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  4. 2024Context-DPO: Aligning Language Models for Context-Faithfulnesscited by 50
  5. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  6. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  7. 2025MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contextscited by 32
  8. 2024"Not Aligned" is Not "Malicious": Being Careful about Hallucinations of Large Language Models' Jailbreakcited by 25
  9. 2025GeoSense: Evaluating Identification and Application of Geometric Principles in Multimodal Reasoningcited by 25
  10. 2024HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Routercited by 16
  11. 2024CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Modelscited by 15
  12. 2025AutoGPS: Automated Geometry Problem Solving via Multimodal Formalization and Deductive Reasoningcited by 14
33 more
  1. 2025a1: Steep Test-time Scaling Law via Environment Augmented Generationcited by 13
  2. 2025TrustGeoGen: Formal-Verified Data Engine for Trustworthy Multi-modal Geometric Problem Solvingcited by 13
  3. 2025A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Sciencecited by 12
  4. 2024Diagram Formalization Enhanced Multi-Modal Geometry Problem Solvercited by 11
  5. 2024HYBRIDMIND: Meta Selection of Natural Language and Symbolic Language for Enhanced LLM Reasoningcited by 11
  6. 2025NeSyGeo: A Neuro-Symbolic Framework for Multimodal Geometric Reasoning Data Generationcited by 10
  7. 2024Understanding Depth and Height Perception in Large Visual-Language Modelscited by 8
  8. 2025Towards Geometry Problem Solving in the Large Model Era: A Surveycited by 8
  9. 2025Do Large Language Models Truly Understand Geometric Structures?cited by 7
  10. 2025VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoningcited by 7
  11. 2025SIRI-Bench: Challenging VLMs' Spatial Intelligence through Complex Reasoning Taskscited by 7
  12. 2025LeanGeo: Formalizing Competitional Geometry problems in Leancited by 7
  13. 2025AgenticMath: Enhancing LLM Reasoning via Agentic-based Math Data Generationcited by 7
  14. 2024MageBench: Bridging Large Multimodal Models to Agentscited by 6
  15. 2025Large Language Models as Computable Approximations to Solomonoff Inductioncited by 6
  16. 2025GeoLaux: A Benchmark for Evaluating MLLMs' Geometry Performance on Long-Step Problems Requiring Auxiliary Linescited by 6
  17. 2025GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Modelscited by 6
  18. 2026Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluationcited by 6
  19. 2026Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representationscited by 5
  20. 2025GeoSDF: Plane Geometry Diagram Synthesis via Signed Distance Fieldcited by 4
  21. 2025A Survey of Deep Learning for Geometry Problem Solvingcited by 4
  22. 2025GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluationcited by 4
  23. 2024Tangram: Benchmark for Evaluating Geometric Element Recognition in Large Multimodal Modelscited by 2
  24. 2025DynaSolidGeo: A Dynamic Benchmark for Genuine Spatial Mathematical Reasoning of VLMs in Solid Geometrycited by 1
  25. 2026GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagramscited by 1
  26. 2026Geoparsing: Diagram Parsing for Plane and Solid Geometry with a Unified Formal Languagecited by 1
  27. 2026FormalAnalyticGeo: A Neural-Symbolic Based Framework for Multimodal Analytic Geometry Problem Generationcited by 1
  28. 2025MathSight: A Benchmark Exploring Have Vision-Language Models Really Seen in University-Level Mathematical Reasoning?cited by 0
  29. 2026BiNSGPS: Geometry Problem Solving via Bidirectional Neuro-Symbolic Interactioncited by 0
  30. 2026Internalizing Geometric Law: Learning from Solver Residuals for Precision-Critical Generationcited by 0
  31. 2026VeriGeo: Controllable Geometry Question Generation with Numerical and Analytical Verificationcited by 0
  32. 2026Euclid-Omni : A Unified Neuro-Symbolic Framework for Plane Geometrycited by 0
  33. 2026Dense Neuro-Symbolic Reasoning in a Unified Geometry Statecited by 0