# Math-Vision Diagrams: A Comprehensive Benchmark for Evaluating LLM Mathematical Diagram Generation Capabilities

arXiv: https://arxiv.org/abs/2608.08964 · HTML: https://mlsanity.com/paper/2608.08964/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 9 Aug 2026.

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://mlsanity.com/paper/2402.14804/) (2024)
- [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://mlsanity.com/paper/2401.09603/) (2023)
- [GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning](https://mlsanity.com/paper/2105.14517/) (2021)
- [OmniSVG: A Unified Scalable Vector Graphics Generation Model](https://mlsanity.com/paper/2504.06263/) (2025)
- [Empowering LLMs to Understand and Generate Complex Vector Graphics](https://mlsanity.com/paper/2412.11102/) (2024)
- [Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Models](https://mlsanity.com/paper/2411.16602/) (2024)
- [MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts](https://mlsanity.com/paper/2502.20808/) (2025)
- [From Words to Structured Visuals: A Benchmark and Framework for Text-to-Diagram Generation and Editing](https://mlsanity.com/paper/2411.11916/) (2024)
- [MMMG: A Massive, Multidisciplinary, Multi-Tier Generation Benchmark for Text-to-Image Reasoning](https://mlsanity.com/paper/2506.10963/) (2025)
- [Draw with Thought: Unleashing Multimodal Reasoning for Scientific Diagram Generation](https://mlsanity.com/paper/2504.09479/) (2025)
- [MagicGeo: Training-Free Text-Guided Geometric Diagram Generation](https://mlsanity.com/paper/2502.13855/) (2025)
- [From Text to Visuals: Using LLMs to Generate Math Diagrams with Vector Graphics](https://mlsanity.com/paper/2503.07429/) (2025)
- [DiagramIR: An Automatic Pipeline for Educational Math Diagram Evaluation](https://mlsanity.com/paper/2511.08283/) (2025)
- [MermaidSeqBench: An Evaluation Benchmark for NL-to-Mermaid Sequence Diagram Generation](https://mlsanity.com/paper/2511.14967/) (2025)
- [GeoLoom: High-quality Geometric Diagram Generation from Textual Input](https://mlsanity.com/paper/2512.08180/) (2025)

## Built on by (most cited first, 1 of 1)

- [MathVis: Can AI Reason through Drawing? A Benchmark for Mathematical Figure Generation](https://mlsanity.com/paper/2610.12954/) (2026)

## Abstract (arXiv)

The generation of mathematically precise diagrams from tex- tual prompts has emerged as a critical yet underexplored capability of Large Language Models (LLMs). This has been of interest to researchers in the areas of curriculum preparation, automated ranking of problem sets, and scientific publishing. For LLMs to achieve this, it requires per- fect coordination between Spatial Reasoning, Mathematical Reasoning, and Rendering systems. While existing benchmarks such as MathVision, MathVista are built for Math Reasoning or DiagramGenBenchmark, Mer- maidSeqBench on general purpose diagram generation, no prior work provides a standardized set of prompt, image pairs that can be used to evaluate the LLMs specifically on math diagram generation. This includes fields that span both both text-to-code and text-to-image paradigms. We introduce Math-Vision Diagrams, the first benchmark specifically designed to evaluate LLMs on mathematical diagram generation, and the first to assess text-to-code and text-to-image generation paradigms together in a single unified setting, agnostic of the underlying coding lan- guage or model type. Building on the Math-Vision benchmark, we select a subset of 2920 images out of 3040 from high-quality competition problems with essential visual context. A novel pipeline combining an ensemble of LLMs with Subject Matter Expert (SME) curation is presented, together with a suite of evaluation metrics. Testing several leading models against this benchmark, we demonstrate that LLMs struggle with math diagram generation. All code, data, curation pipeline, and evaluation scripts will be fully open-sourced.