# From Words to Structured Visuals: A Benchmark and Framework for Text-to-Diagram Generation and Editing

arXiv: https://arxiv.org/abs/2411.11916 · HTML: https://mlsanity.com/paper/2411.11916/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Nov 2024.

## Built on by (most cited first, 30 of 31)

- [Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation](https://mlsanity.com/paper/2502.05151/) (2025)
- [Code2Video: A Code-centric Paradigm for Educational Video Generation](https://mlsanity.com/paper/2510.01174/) (2025)
- [AutoFigure: Generating and Refining Publication-Ready Scientific Illustrations](https://mlsanity.com/paper/2602.03828/) (2026)
- [AutoFigure-Edit: Generating Editable Scientific Illustration](https://mlsanity.com/paper/2603.06674/) (2026)
- [Geoint-R1: Formalizing Multimodal Geometric Reasoning with Dynamic Auxiliary Constructions](https://mlsanity.com/paper/2508.03173/) (2025)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editing](https://mlsanity.com/paper/2605.15677/) (2026)
- [Automatic Method Illustration Generation for AI Scientific Papers via Drawing Middleware Creation, Evolution, and Orchestration](https://mlsanity.com/paper/2603.29590/) (2026)
- [SciFigDetect: A Benchmark for AI-Generated Scientific Figure Detection](https://mlsanity.com/paper/2604.08211/) (2026)
- [Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Tasks](https://mlsanity.com/paper/2604.19697/) (2026)
- [SketchAgent: Generating Structured Diagrams from Hand-Drawn Sketches](https://mlsanity.com/paper/2508.01237/) (2025)
- [OpusAnimation: Code-Based Dynamic Chart Generation](https://mlsanity.com/paper/2510.03341/) (2025)
- [ResearchPulse: Building Method-Experiment Chains through Multi-Document Scientific Inference](https://mlsanity.com/paper/2509.03565/) (2025)
- [GeoLoom: High-quality Geometric Diagram Generation from Textual Input](https://mlsanity.com/paper/2512.08180/) (2025)
- [TechImage-Bench: Rubric-Based Evaluation for Technical Image Generation](https://mlsanity.com/paper/2512.12220/) (2025)
- [TechING: Towards Real World Technical Image Understanding via VLMs](https://mlsanity.com/paper/2601.18238/) (2026)
- [EvoDiagram: Agentic Editable Diagram Creation via Design Expertise Evolution](https://mlsanity.com/paper/2604.09568/) (2026)
- [VisDocSketcher: Towards Scalable Visual Documentation with Agentic Systems](https://mlsanity.com/paper/2509.11942/) (2025)
- [TeachMaster: Generative Teaching via Code](https://mlsanity.com/paper/2601.04204/) (2025)
- [Text2Arch: A Dataset for Generating Scientific Architecture Diagrams from Natural Language Descriptions](https://mlsanity.com/paper/2604.14941/) (2026)
- [PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents](https://mlsanity.com/paper/2605.10341/) (2026)
- [PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning](https://mlsanity.com/paper/2605.19382/) (2026)
- [SciForma: Structure-Faithful Generation of Scientific Diagrams](https://mlsanity.com/paper/2607.18091/) (2026)
- [Math-Vision Diagrams: A Comprehensive Benchmark for Evaluating LLM Mathematical Diagram Generation Capabilities](https://mlsanity.com/paper/2608.08964/) (2026)
- [EdiTikZ: Scientific Figure Editing from Revision Trajectories](https://mlsanity.com/paper/2609.01409/) (2026)
- [CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval](https://mlsanity.com/paper/2604.15663/) (2026)
- [Faithful, Enriched, and Precise: Benchmarking Natural-Science Illustration Generation by T2I models](https://mlsanity.com/paper/2606.05949/) (2026)
- [GenGA: Editable and Data-Grounded Graphical Abstract Generation for Academic Papers](https://mlsanity.com/paper/2608.05478/) (2026)
- [ReFigBench: Benchmarking Scientific Figure Reconstruction as Editable PowerPoint Artifacts](https://mlsanity.com/paper/2609.18844/) (2026)
- [FigAct: Turning Scientific Figures into Active Canvases for Explanation](https://mlsanity.com/paper/2609.36190/) (2026)

## Abstract (arXiv)

We introduce the task of text-to-diagram generation, which focuses on creating structured visual representations directly from textual descriptions. Existing approaches in text-to-image and text-to-code generation lack the logical organization and flexibility needed to produce accurate, editable diagrams, often resulting in outputs that are either unstructured or difficult to modify. To address this gap, we introduce DiagramGenBenchmark, a comprehensive evaluation framework encompassing eight distinct diagram categories, including flowcharts, model architecture diagrams, and mind maps. Additionally, we present DiagramAgent, an innovative framework with four core modules-Plan Agent, Code Agent, Check Agent, and Diagram-to-Code Agent-designed to facilitate both the generation and refinement of complex diagrams. Our extensive experiments, which combine objective metrics with human evaluations, demonstrate that DiagramAgent significantly outperforms existing baseline models in terms of accuracy, structural coherence, and modifiability. This work not only establishes a foundational benchmark for the text-to-diagram generation task but also introduces a powerful toolset to advance research and applications in this emerging area.