# Why LLMs Are Bad at Synthetic Table Generation (and what to do about it)

arXiv: https://arxiv.org/abs/2406.14541 · HTML: https://mlsanity.com/paper/2406.14541/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 51 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [Language Models are Realistic Tabular Data Generators](https://mlsanity.com/paper/2210.06280/) (2022)
- [Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space](https://mlsanity.com/paper/2310.09656/) (2023)
- [Data Synthesis based on Generative Adversarial Networks](https://mlsanity.com/paper/1806.03384/) (2018)
- [LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks](https://mlsanity.com/paper/2206.06565/) (2022)
- [Why think step by step? Reasoning emerges from the locality of experience](https://mlsanity.com/paper/2304.03843/) (2023)
- [REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers](https://mlsanity.com/paper/2302.02041/) (2023)
- [Synthesizing Tabular Data using Generative Adversarial Networks](https://mlsanity.com/paper/1811.11264/) (2018)
- [Generative Table Pre-training Empowers Models for Tabular Prediction](https://mlsanity.com/paper/2305.09696/) (2023)
- [Supervised Learning on Relational Databases with Graph Neural Networks](https://mlsanity.com/paper/2002.02046/) (2020)
- [TabuLa: Harnessing Language Models for Tabular Data Synthesis](https://mlsanity.com/paper/2310.12746/) (2023)
- [TabularNet: A Neural Network Architecture for Understanding Semantic Structures of Tabular Data](https://mlsanity.com/paper/2106.03096/) (2021)
- [Enhancing AMR-to-Text Generation with Dual Graph Representations](https://mlsanity.com/paper/1909.00352/) (2019)
- [STAN: Synthetic Network Traffic Generation with Generative Neural Models](https://mlsanity.com/paper/2009.12740/) (2020)

## Built on by (most cited first, 10 of 10)

- [Doubling Your Data in Minutes: Ultra-fast Tabular Data Generation via LLM-Induced Dependency Graphs](https://mlsanity.com/paper/2507.19334/) (2025)
- [Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Models](https://mlsanity.com/paper/2507.18504/) (2025)
- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)
- [TREB: a BERT attempt for imputing tabular data imputation](https://mlsanity.com/paper/2410.00022/) (2024)
- [Making Sense of Data in the Wild: Data Analysis Automation at Scale](https://mlsanity.com/paper/2502.15718/) (2025)
- [SAGE: Sparse Adaptive Guidance for Dependency-Aware Tabular Data Generation](https://mlsanity.com/paper/2604.24368/) (2026)
- [A Note on Statistically Accurate Tabular Data Generation Using Large Language Models](https://mlsanity.com/paper/2505.02659/) (2025)
- [StructSynth: Dependency Graphs as Generation Plans for Low-Data Tabular Synthesis with Language Models](https://mlsanity.com/paper/2508.02601/) (2025)
- [Declarative by Design, Assistable Only by Convention: Benchmarking Multi-Agent Frameworks for AI-Assistability](https://mlsanity.com/paper/2602.11198/) (2026)
- [Scalable extraction and visualization of multi-attribute logical and functional dependencies in tabular data](https://mlsanity.com/paper/2610.08287/) (2026)

## Abstract (arXiv)

Synthetic data generation is integral to ML pipelines, e.g., to augment training data, replace sensitive information, and even to power advanced platforms like DeepSeek. While LLMs fine-tuned for synthetic data generation are gaining traction, synthetic table generation -- a critical data type in business and science -- remains under-explored compared to text and image synthesis. This paper shows that LLMs, whether used as-is or after traditional fine-tuning, are inadequate for generating synthetic tables. Their autoregressive nature, combined with random order permutation during fine-tuning, hampers the modeling of functional dependencies and prevents capturing conditional mixtures of distributions essential for real-world constraints. We demonstrate that making LLMs permutation-aware can mitigate these issues.