# GeoEval: Benchmark for Evaluating LLMs and Multi-Modal Models on Geometry Problem-Solving

arXiv: https://arxiv.org/abs/2402.10104 · HTML: https://mlsanity.com/paper/2402.10104/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 15 Feb 2024.

## Builds on (most cited first, 21 of 21)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning](https://mlsanity.com/paper/2105.04165/) (2021)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [Evaluating the Performance of Large Language Models on GAOKAO Benchmark](https://mlsanity.com/paper/2305.12474/) (2023)
- [VLP: A Survey on Vision-Language Pre-training](https://mlsanity.com/paper/2202.09061/) (2022)
- [UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression](https://mlsanity.com/paper/2212.02746/) (2022)
- [GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoning](https://mlsanity.com/paper/2312.12241/) (2023)
- [A Multi-Modal Neural Geometric Solver with Textual Clauses Parsed from Diagram](https://mlsanity.com/paper/2302.11097/) (2023)
- [Forward-Backward Reasoning in Large Language Models for Mathematical Verification](https://mlsanity.com/paper/2308.07758/) (2023)
- [LANS: A Layout-Aware Neural Solver for Plane Geometry Problem](https://mlsanity.com/paper/2311.16476/) (2023)
- [Sequence to General Tree: Knowledge-Guided Geometry Word Problem Solving](https://mlsanity.com/paper/2106.00990/) (2021)
- [GAPS: Geometry-Aware Problem Solver](https://mlsanity.com/paper/2401.16287/) (2024)

## Built on by (most cited first, 30 of 45)

- [Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents](https://mlsanity.com/paper/2503.24047/) (2025)
- [Evaluating Step-by-step Reasoning Traces: A Survey](https://mlsanity.com/paper/2502.12289/) (2025)
- [A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges](https://mlsanity.com/paper/2412.11936/) (2024)
- [Context-DPO: Aligning Language Models for Context-Faithfulness](https://mlsanity.com/paper/2412.15280/) (2024)
- [ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection](https://mlsanity.com/paper/2410.04509/) (2024)
- [Visual Large Language Models for Generalized and Specialized Applications](https://mlsanity.com/paper/2501.02765/) (2025)
- [MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts](https://mlsanity.com/paper/2502.20808/) (2025)
- ["Not Aligned" is Not "Malicious": Being Careful about Hallucinations of Large Language Models' Jailbreak](https://mlsanity.com/paper/2406.11668/) (2024)
- [GeoSense: Evaluating Identification and Application of Geometric Principles in Multimodal Reasoning](https://mlsanity.com/paper/2504.12597/) (2025)
- [HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Router](https://mlsanity.com/paper/2410.02684/) (2024)
- [CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models](https://mlsanity.com/paper/2407.12023/) (2024)
- [AutoGPS: Automated Geometry Problem Solving via Multimodal Formalization and Deductive Reasoning](https://mlsanity.com/paper/2505.23381/) (2025)
- [a1: Steep Test-time Scaling Law via Environment Augmented Generation](https://mlsanity.com/paper/2504.14597/) (2025)
- [TrustGeoGen: Formal-Verified Data Engine for Trustworthy Multi-modal Geometric Problem Solving](https://mlsanity.com/paper/2504.15780/) (2025)
- [A Survey of Large Language Model-Powered Spatial Intelligence Across Scales: Advances in Embodied Agents, Smart Cities, and Earth Science](https://mlsanity.com/paper/2504.09848/) (2025)
- [Diagram Formalization Enhanced Multi-Modal Geometry Problem Solver](https://mlsanity.com/paper/2409.04214/) (2024)
- [HYBRIDMIND: Meta Selection of Natural Language and Symbolic Language for Enhanced LLM Reasoning](https://mlsanity.com/paper/2409.19381/) (2024)
- [NeSyGeo: A Neuro-Symbolic Framework for Multimodal Geometric Reasoning Data Generation](https://mlsanity.com/paper/2505.17121/) (2025)
- [Understanding Depth and Height Perception in Large Visual-Language Models](https://mlsanity.com/paper/2408.11748/) (2024)
- [Towards Geometry Problem Solving in the Large Model Era: A Survey](https://mlsanity.com/paper/2506.02690/) (2025)
- [Do Large Language Models Truly Understand Geometric Structures?](https://mlsanity.com/paper/2501.13773/) (2025)
- [VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning](https://mlsanity.com/paper/2506.02537/) (2025)
- [SIRI-Bench: Challenging VLMs' Spatial Intelligence through Complex Reasoning Tasks](https://mlsanity.com/paper/2506.14512/) (2025)
- [LeanGeo: Formalizing Competitional Geometry problems in Lean](https://mlsanity.com/paper/2508.14644/) (2025)
- [AgenticMath: Enhancing LLM Reasoning via Agentic-based Math Data Generation](https://mlsanity.com/paper/2510.19361/) (2025)
- [MageBench: Bridging Large Multimodal Models to Agents](https://mlsanity.com/paper/2412.04531/) (2024)
- [Large Language Models as Computable Approximations to Solomonoff Induction](https://mlsanity.com/paper/2505.15784/) (2025)
- [GeoLaux: A Benchmark for Evaluating MLLMs' Geometry Performance on Long-Step Problems Requiring Auxiliary Lines](https://mlsanity.com/paper/2508.06226/) (2025)
- [GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Models](https://mlsanity.com/paper/2511.11134/) (2025)
- [Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluation](https://mlsanity.com/paper/2602.11635/) (2026)

## Abstract (arXiv)

Recent advancements in large language models (LLMs) and multi-modal models (MMs) have demonstrated their remarkable capabilities in problem-solving. Yet, their proficiency in tackling geometry math problems, which necessitates an integrated understanding of both textual and visual information, has not been thoroughly evaluated. To address this gap, we introduce the GeoEval benchmark, a comprehensive collection that includes a main subset of 2,000 problems, a 750 problems subset focusing on backward reasoning, an augmented subset of 2,000 problems, and a hard subset of 300 problems. This benchmark facilitates a deeper investigation into the performance of LLMs and MMs in solving geometry math problems. Our evaluation of ten LLMs and MMs across these varied subsets reveals that the WizardMath model excels, achieving a 55.67% accuracy rate on the main subset but only a 6.00% accuracy on the hard subset. This highlights the critical need for testing models against datasets on which they have not been pre-trained. Additionally, our findings indicate that GPT-series models perform more effectively on problems they have rephrased, suggesting a promising method for enhancing model capabilities.