mlsanityData to 14 Oct 2026

GeoGramBench: Benchmarking the Geometric Program Reasoning in Modern LLMs

arXiv 2505.17653 ↗First posted 23 May 2025cs.AIcited by 13

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Geometric spatial reasoning forms the foundation of many applications in artificial intelligence, yet the ability of large language models (LLMs) to operate over geometric spatial information expressed in procedural code remains underexplored. In this paper, we address this gap by formalizing the Program-to-Geometry task, which challenges models to translate programmatic drawing code into accurate and abstract geometric reasoning. To evaluate this capability, we present GeoGramBench, a benchmark of 500 carefully refined problems organized by a tailored three-level taxonomy that considers geometric complexity rather than traditional mathematical reasoning complexity. Our comprehensive evaluation of 17 frontier LLMs reveals consistent and pronounced deficiencies: even the most advanced models achieve less than 50% accuracy at the highest abstraction level. These results highlight the unique challenges posed by program-driven spatial reasoning and establish GeoGramBench as a valuable resource for advancing research in symbolic-to-spatial geometric reasoning. Project page: https://github.com/LiAuto-DSR/GeoGramBench.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 43 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

28 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025GeoSense: Evaluating Identification and Application of Geometric Principles in Multimodal Reasoning17 Apr
  2. 2025LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?25 Mar
  3. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec

Other references · 25

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  9. 2025s1: Simple test-time scalingcited by 1,341
  10. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  11. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  12. 2025LIMO: Less is More for Reasoningcited by 498
13 more
  1. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  2. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 201
  3. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  4. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 109
  5. 2025S*: Test Time Scaling for Code Generationcited by 98
  6. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Modelscited by 70
  7. 2024Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Modelscited by 38
  8. 2024MM-MATH: Advancing Multimodal Math Evaluation with Process Evaluation and Fine-grained Classificationcited by 35
  9. 2024Can Large Language Models Understand Symbolic Graphics Programs?cited by 35
  10. 2024HARP: A challenging human-annotated math reasoning benchmarkcited by 27
  11. 2025Understanding and Benchmarking Artificial Intelligence: OpenAI's o3 Is Not AGIcited by 19
  12. 2024Exploring the Capability of LLMs in Performing Low-Level Visual Analytic Tasks on SVG Data Visualizationscited by 18
  13. 2024Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptionscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

By area:

  1. 2026Code as Agent Harnesscited by 47
  2. 2025BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Constructioncited by 9
  3. 2026SpatialEvo: Self-Evolving Spatial Intelligence via Deterministic Geometric Environmentscited by 5
  4. 2026GPSBench: Do Large Language Models Understand GPS Coordinates?cited by 4
  5. 2025GDGS: 3D Gaussian Splatting Via Geometry-Guided Initialization And Dynamic Density Controlcited by 2
  6. 2026NoReGeo: Non-Reasoning Geometry Benchmarkcited by 2
  7. 2026OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documentscited by 2
  8. 2025A Global-Local Cross-Attention Network for Ultra-high Resolution Remote Sensing Image Semantic Segmentationcited by 1
  9. 2026GeoChallenge: A Multi-Answer Multiple-Choice Benchmark for Geometric Reasoning with Diagramscited by 1
  10. 2026PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoningcited by 1
  11. 2025FractalBench: Diagnosing Visual-Mathematical Reasoning Through Recursive Program Synthesiscited by 0
  12. 2026Draw2Think: Harnessing Geometry Reasoning through Constraint Engine Interactioncited by 0
1 more
  1. 2026HALO: A Physics-Aware LLM Agent Framework for Nanophotonic Designcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Geoint-R1: Formalizing Multimodal Geometric Reasoning with Dynamic Auxiliary Constructions
  2. 2025GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluation
  3. 2026From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models
  4. 2025GeoFM: Enhancing Geometric Reasoning of MLLMs via Synthetic Data Generation through Formal Language
  5. 2024GeoCoder: Solving Geometry Problems by Generating Modular Code through Vision-Language Models
  6. 2025Towards Geometry Problem Solving in the Large Model Era: A Survey
  7. 2025GeoLaux: A Benchmark for Evaluating MLLMs' Geometry Performance on Long-Step Problems Requiring Auxiliary Lines
  8. 2023GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoning