# Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data?

arXiv: https://arxiv.org/abs/2309.08963 · HTML: https://mlsanity.com/paper/2309.08963/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 16 Sep 2023.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!](https://mlsanity.com/paper/2303.08559/) (2023)
- [Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method](https://mlsanity.com/paper/2305.13412/) (2023)
- [Text-to-Table: A New Way of Information Extraction](https://mlsanity.com/paper/2109.02707/) (2021)
- [RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations](https://mlsanity.com/paper/2306.14321/) (2023)
- [A Sequence-to-Sequence&Set Model for Text-to-Table Generation](https://mlsanity.com/paper/2306.00137/) (2023)
- [Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries](https://mlsanity.com/paper/2109.09195/) (2021)
- [STable: Table Generation Framework for Encoder-Decoder Models](https://mlsanity.com/paper/2206.04045/) (2022)
- [KnowGL: Knowledge Generation and Linking from Text](https://mlsanity.com/paper/2210.13952/) (2022)
- [Revisiting Event Argument Extraction: Can EAE Models Learn Better When Being Aware of Event Co-occurrences?](https://mlsanity.com/paper/2306.00502/) (2023)
- [WebIE: Faithful and Robust Information Extraction on the Web](https://mlsanity.com/paper/2305.14293/) (2023)

## Built on by (most cited first, 16 of 16)

- [CFBench: A Comprehensive Constraints-Following Benchmark for LLMs](https://mlsanity.com/paper/2408.01122/) (2024)
- [RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios](https://mlsanity.com/paper/2412.08972/) (2024)
- [StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs](https://mlsanity.com/paper/2505.20139/) (2025)
- [Classroom Simulacra: Building Contextual Student Generative Agents in Online Education for Learning Behavioral Simulation](https://mlsanity.com/paper/2502.02780/) (2025)
- [StrucText-Eval: Evaluating Large Language Model's Reasoning Ability in Structure-Rich Text](https://mlsanity.com/paper/2406.10621/) (2024)
- [ReFF: Reinforcing Format Faithfulness in Language Models across Varied Tasks](https://mlsanity.com/paper/2412.09173/) (2024)
- [EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Models](https://mlsanity.com/paper/2506.08375/) (2025)
- [AIC CTU system at AVeriTeC: Re-framing automated fact-checking as a simple RAG task](https://mlsanity.com/paper/2410.11446/) (2024)
- [MT-RAIG: Novel Benchmark and Evaluation Framework for Retrieval-Augmented Insight Generation over Multiple Tables](https://mlsanity.com/paper/2502.11735/) (2025)
- [TeXpert: A Multi-Level Benchmark for Evaluating LaTeX Code Generation by LLMs](https://mlsanity.com/paper/2506.16990/) (2025)
- [SO-Bench: A Structural Output Evaluation of Multimodal LLMs](https://mlsanity.com/paper/2511.21750/) (2025)
- [Intent-Aware Schema Generation And Refinement For Literature Review Tables](https://mlsanity.com/paper/2507.19521/) (2025)
- [iTBLS: A Dataset of Interactive Conversations Over Tabular Information](https://mlsanity.com/paper/2404.12580/) (2024)
- [DEGAP: Dual Event-Guided Adaptive Prefixes for Templated-Based Event Argument Extraction with Slot Querying](https://mlsanity.com/paper/2405.13325/) (2024)
- [A Layered Architecture for Developing and Enhancing Capabilities in Large Language Model-based Software Systems](https://mlsanity.com/paper/2411.12357/) (2024)
- [Towards Efficient and Effective Alignment of Large Language Models](https://mlsanity.com/paper/2506.09329/) (2025)

## Abstract (arXiv)

Despite the remarkable capabilities of Large Language Models (LLMs) like GPT-4, producing complex, structured tabular data remains challenging. Our study assesses LLMs' proficiency in structuring tables and introduces a novel fine-tuning method, cognizant of data structures, to bolster their performance. We unveil Struc-Bench, a comprehensive benchmark featuring prominent LLMs (GPT-NeoX-20B, GPT-3.5, GPT-4, and Vicuna), which spans text tables, HTML, and LaTeX formats. Our proposed FormatCoT aids in crafting format-specific instructions from the intended outputs to populate this benchmark. Addressing the gap in task-centered evaluation, we propose two innovative metrics, P-Score (Prompting Score) and H-Score (Heuristical Score), to more accurately gauge LLM performance. Our experiments show that applying our structure-aware fine-tuning to LLaMA-7B leads to substantial performance gains, outshining its LLM counterparts across most measures. In-depth error analysis and creating an ability map across six dimensions -- coverage, formatting, reasoning, comprehension, pragmatics, and hallucination -- highlight areas for future enhancements and suggest forthcoming research trajectories. Our code and models can be found at https://github.com/gersteinlab/Struc-Bench.