mlsanityData to 7 Oct 2026

StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs

arXiv 2505.20139 ↗First posted 26 May 2025cs.SE cs.AI cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

As Large Language Models (LLMs) become integral to software development workflows, their ability to generate structured outputs has become critically important. We introduce StructEval, a comprehensive benchmark for evaluating LLMs' capabilities in producing both non-renderable (JSON, YAML, CSV) and renderable (HTML, React, SVG) structured formats. Unlike prior benchmarks, StructEval systematically evaluates structural fidelity across diverse formats through two paradigms: 1) generation tasks, producing structured output from natural language prompts, and 2) conversion tasks, translating between structured formats. Our benchmark encompasses 18 formats and 44 types of task, with novel metrics for format adherence and structural correctness. Results reveal significant performance gaps-even state-of-the-art models like o1-mini achieve only 75.58 average score, with open-source alternatives lagging approximately 10 points behind. We find generation tasks more challenging than conversion tasks, and producing correct visual content more difficult than generating text-only structures.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

23 of 34 references.

References · 23

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  7. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  9. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  10. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  11. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  12. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 146
11 more
  1. 2023StarVector: Generating Scalable Vector Graphics Code from Images and Textcited by 96
  2. 2024FOFO: A Benchmark to Evaluate LLMs' Format-Following Capabilitycited by 79
  3. 2023Table-GPT: Table-tuned GPT for Diverse Table Taskscited by 69
  4. 2024TableGPT2: A Large Multimodal Model with Tabular Data Integrationcited by 69
  5. 2023GPT-RE: In-context Learning for Relation Extraction using Large Language Modelscited by 62
  6. 2023Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data?cited by 16
  7. 2024PiCO: Peer Review in LLMs based on the Consistency Optimizationcited by 14
  8. 2024StackEval: Benchmarking LLMs in Coding Assistancecited by 14
  9. 2024StrucText-Eval: Evaluating Large Language Model's Reasoning Ability in Structure-Rich Textcited by 13
  10. 2024OverleafCopilot: Empowering Academic Writing in Overleaf with Large Language Modelscited by 7
  11. 2025From Text to Visuals: Using LLMs to Generate Math Diagrams with Vector Graphicscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 16.

23 papers.

Sort:

By area:

  1. 2025BrowserAgent: Building Web Agents with Human-Inspired Web Browsing Actionscited by 17
  2. 2026LLMs Corrupt Your Documents When You Delegatecited by 9
  3. 2026ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extractioncited by 8
  4. 2026AutoPyVerifier: Learning Compact Executable Verifiers for Large Language Model Outputscited by 7
  5. 2025SO-Bench: A Structural Output Evaluation of Multimodal LLMscited by 5
  6. 2025STED and Consistency Scoring: A Framework for Evaluating LLM Structured Output Reliabilitycited by 5
  7. 2026Evaluating Robustness of Large Language Models in Enterprise Applications: Benchmarks for Perturbation Consistency Across Formats and Languagescited by 4
  8. 2026The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Modelscited by 3
  9. 2025Vector Prism: Animating Vector Graphics by Stratifying Semantic Structurecited by 2
  10. 2025Performance Trade-offs of Optimizing Small Language Models for E-Commercecited by 1
  11. 2026FMBench: Adaptive Large Language Model Output Formattingcited by 1
  12. 2026Large Language Models for Geolocation Extraction in Humanitarian Crisis Responsecited by 1
11 more
  1. 2026T2S-Bench & Structure-of-Thought: Benchmarking and Prompting Comprehensive Text-to-Structure Reasoningcited by 1
  2. 2025Benchmarking Agentic Newswriting via Journalistic Workflowscited by 0
  3. 2025DeepJSONEval: Benchmarking Complex Nested JSON Data Mining for Large Language Modelscited by 0
  4. 2026Generating Hierarchical JSON Representations of Scientific Sentences Using LLMscited by 0
  5. 2026Accelerating Constrained Decoding with Token Space Compressioncited by 0
  6. 2026A Hybrid LLM-Based Framework for Automated Security Annotation Generation in Business Process Modelscited by 0
  7. 2026TRACE-BN: Transferring Bangla-English Tutoring Behavior to a Sub-1B Offline Language Modelcited by 0
  8. 2026LatentMD: Benchmarking Markdown Boundary Failures in LLM-Generated Textcited by 0
  9. 2026MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimescited by 0
  10. 2026NeMo Data Designer: An Extensible Framework for Multimodal Synthetic Data Generationcited by 0
  11. 2026EvidenT: Building Trustworthy Enterprise Assistants through Evidence Groundedness and Traceabilitycited by 0