# DiagramEval: Evaluating LLM-Generated Diagrams via Graphs

arXiv: https://arxiv.org/abs/2510.25761 · HTML: https://mlsanity.com/paper/2510.25761/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 44 references that are in the map.
- First posted on arXiv 29 Oct 2025.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Can Language Models Solve Graph Problems in Natural Language?](https://mlsanity.com/paper/2305.10037/) (2023)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning](https://mlsanity.com/paper/2311.10774/) (2023)
- [GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph](https://mlsanity.com/paper/2105.02605/) (2021)
- [AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ](https://mlsanity.com/paper/2310.00367/) (2023)
- [FIRST: A Framework for Optimizing Information Quality in Mobile Crowdsensing Systems](https://mlsanity.com/paper/1804.11147/) (2018)
- [OCR-VQGAN: Taming Text-within-Image Generation](https://mlsanity.com/paper/2210.11248/) (2022)
- [DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning](https://mlsanity.com/paper/2310.12128/) (2023)
- [SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinement](https://mlsanity.com/paper/2409.19242/) (2024)
- [Draw with Thought: Unleashing Multimodal Reasoning for Scientific Diagram Generation](https://mlsanity.com/paper/2504.09479/) (2025)

## Built on by (most cited first, 8 of 8)

- [PaperBanana: Automating Academic Illustration for AI Scientists](https://mlsanity.com/paper/2601.23265/) (2026)
- [Paper2SysArch: Structure-Constrained System Architecture Generation from Scientific Papers](https://mlsanity.com/paper/2511.18036/) (2025)
- [Automatic Method Illustration Generation for AI Scientific Papers via Drawing Middleware Creation, Evolution, and Orchestration](https://mlsanity.com/paper/2603.29590/) (2026)
- [GeoSVG-RL: Geometry-Aware Reinforcement Learning for Layout-Constrained Text-to-SVG Diagram Generation](https://mlsanity.com/paper/2605.25447/) (2026)
- [PPTBench: Can Coding Agents Reconstruct the Visual World through Structured, Editable Slides](https://mlsanity.com/paper/2609.29718/) (2026)
- [Text to Automata Diagrams: Comparing TikZ Code Generation with Direct Image Synthesis](https://mlsanity.com/paper/2603.07936/) (2026)
- [DiagramRAG: A Lightweight Framework to Retrieve Scientific Diagram for Figure Generation](https://mlsanity.com/paper/2605.27931/) (2026)
- [One Figure, Every Canvas: Editable Flowchart Relayout via Agentic Pipeline](https://mlsanity.com/paper/2610.06852/) (2026)

## Abstract (arXiv)

Diagrams play a central role in research papers for conveying ideas, yet they are often notoriously complex and labor-intensive to create. Although diagrams are presented as images, standard image generative models struggle to produce clear diagrams with well-defined structure. We argue that a promising direction is to generate demonstration diagrams directly in textual form as SVGs, which can leverage recent advances in large language models (LLMs). However, due to the complexity of components and the multimodal nature of diagrams, sufficiently discriminative and explainable metrics for evaluating the quality of LLM-generated diagrams remain lacking. In this paper, we propose DiagramEval, a novel evaluation metric designed to assess demonstration diagrams generated by LLMs. Specifically, DiagramEval conceptualizes diagrams as graphs, treating text elements as nodes and their connections as directed edges, and evaluates diagram quality using two new groups of metrics: node alignment and path alignment. For the first time, we effectively evaluate diagrams produced by state-of-the-art LLMs on recent research literature, quantitatively demonstrating the validity of our metrics. Furthermore, we show how the enhanced explainability of our proposed metrics offers valuable insights into the characteristics of LLM-generated diagrams. Code: https://github.com/ulab-uiuc/diagram-eval.