# StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs

arXiv: https://arxiv.org/abs/2505.20139 · HTML: https://mlsanity.com/paper/2505.20139/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 23 of 23)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering](https://mlsanity.com/paper/2403.03163/) (2024)
- [StarVector: Generating Scalable Vector Graphics Code from Images and Text](https://mlsanity.com/paper/2312.11556/) (2023)
- [FOFO: A Benchmark to Evaluate LLMs' Format-Following Capability](https://mlsanity.com/paper/2402.18667/) (2024)
- [TableGPT2: A Large Multimodal Model with Tabular Data Integration](https://mlsanity.com/paper/2411.02059/) (2024)
- [Table-GPT: Table-tuned GPT for Diverse Table Tasks](https://mlsanity.com/paper/2310.09263/) (2023)
- [GPT-RE: In-context Learning for Relation Extraction using Large Language Models](https://mlsanity.com/paper/2305.02105/) (2023)
- [Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data?](https://mlsanity.com/paper/2309.08963/) (2023)
- [StackEval: Benchmarking LLMs in Coding Assistance](https://mlsanity.com/paper/2412.05288/) (2024)
- [PiCO: Peer Review in LLMs based on the Consistency Optimization](https://mlsanity.com/paper/2402.01830/) (2024)
- [StrucText-Eval: Evaluating Large Language Model's Reasoning Ability in Structure-Rich Text](https://mlsanity.com/paper/2406.10621/) (2024)
- [OverleafCopilot: Empowering Academic Writing in Overleaf with Large Language Models](https://mlsanity.com/paper/2403.09733/) (2024)
- [From Text to Visuals: Using LLMs to Generate Math Diagrams with Vector Graphics](https://mlsanity.com/paper/2503.07429/) (2025)

## Built on by (most cited first, 25 of 25)

- [BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actions](https://mlsanity.com/paper/2510.10666/) (2025)
- [ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction](https://mlsanity.com/paper/2602.12247/) (2026)
- [LLMs Corrupt Your Documents When You Delegate](https://mlsanity.com/paper/2604.15597/) (2026)
- [AutoPyVerifier: Learning Compact Executable Verifiers for Large Language Model Outputs](https://mlsanity.com/paper/2604.22937/) (2026)
- [SO-Bench: A Structural Output Evaluation of Multimodal LLMs](https://mlsanity.com/paper/2511.21750/) (2025)
- [STED and Consistency Scoring: A Framework for Evaluating LLM Structured Output Reliability](https://mlsanity.com/paper/2512.23712/) (2025)
- [Evaluating Robustness of Large Language Models in Enterprise Applications: Benchmarks for Perturbation Consistency Across Formats and Languages](https://mlsanity.com/paper/2601.06341/) (2026)
- [The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models](https://mlsanity.com/paper/2604.25359/) (2026)
- [Vector Prism: Animating Vector Graphics by Stratifying Semantic Structure](https://mlsanity.com/paper/2512.14336/) (2025)
- [T2S-Bench & Structure-of-Thought: Benchmarking and Prompting Comprehensive Text-to-Structure Reasoning](https://mlsanity.com/paper/2603.03790/) (2026)
- [Performance Trade-offs of Optimizing Small Language Models for E-Commerce](https://mlsanity.com/paper/2510.21970/) (2025)
- [FMBench: Adaptive Large Language Model Output Formatting](https://mlsanity.com/paper/2602.06384/) (2026)
- [Large Language Models for Geolocation Extraction in Humanitarian Crisis Response](https://mlsanity.com/paper/2602.08872/) (2026)
- [NeMo Data Designer: An Extensible Framework for Multimodal Synthetic Data Generation](https://mlsanity.com/paper/2609.17699/) (2026)
- [Benchmarking Agentic Newswriting via Journalistic Workflows](https://mlsanity.com/paper/2509.00446/) (2025)
- [DeepJSONEval: Benchmarking Complex Nested JSON Data Mining for Large Language Models](https://mlsanity.com/paper/2509.25922/) (2025)
- [Generating Hierarchical JSON Representations of Scientific Sentences Using LLMs](https://mlsanity.com/paper/2603.23532/) (2026)
- [Accelerating Constrained Decoding with Token Space Compression](https://mlsanity.com/paper/2605.29986/) (2026)
- [Solve the Missing First Step: Can VLMs Standardize Raw Heterogeneous Medical Data?](https://mlsanity.com/paper/2607.04694/) (2026)
- [A Hybrid LLM-Based Framework for Automated Security Annotation Generation in Business Process Models](https://mlsanity.com/paper/2608.14370/) (2026)
- [TRACE-BN: Transferring Bangla-English Tutoring Behavior to a Sub-1B Offline Language Model](https://mlsanity.com/paper/2608.15223/) (2026)
- [LatentMD: Benchmarking Markdown Boundary Failures in LLM-Generated Text](https://mlsanity.com/paper/2609.06993/) (2026)
- [MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimes](https://mlsanity.com/paper/2609.10016/) (2026)
- [EvidenT: Building Trustworthy Enterprise Assistants through Evidence Groundedness and Traceability](https://mlsanity.com/paper/2609.22537/) (2026)
- [ReFormBench: Benchmarking the Stability of Response-Conditioned Format Correction in Large Language Models](https://mlsanity.com/paper/2610.19693/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) become integral to software development workflows, their ability to generate structured outputs has become critically important. We introduce StructEval, a comprehensive benchmark for evaluating LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks, StructEval systematically evaluates structural fidelity across diverse formats through two paradigms: 1) generation tasks, producing structured output from natural language prompts, and 2) conversion tasks, translating between structured formats. Our benchmark encompasses 18 formats and 44 types of task, with novel metrics for format adherence and structural correctness. Results reveal significant performance gaps-even state-of-the-art models like o1-mini achieve only 75.58 average score, with open-source alternatives lagging approximately 10 points behind. We find generation tasks more challenging than conversion tasks, and producing correct visual content more difficult than generating text-only structures.