# Language Models are Realistic Tabular Data Generators

arXiv: https://arxiv.org/abs/2210.06280 · HTML: https://mlsanity.com/paper/2210.06280/

## Key facts

- Cited by 294 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 59 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [Generating Multi-label Discrete Patient Records using Generative Adversarial Networks](https://mlsanity.com/paper/1703.06490/) (2017)
- [Data Synthesis based on Generative Adversarial Networks](https://mlsanity.com/paper/1806.03384/) (2018)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks](https://mlsanity.com/paper/2206.06565/) (2022)
- [Learning Model-Agnostic Counterfactual Explanations for Tabular Data](https://mlsanity.com/paper/1910.09398/) (2019)
- [Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning](https://mlsanity.com/paper/2106.02584/) (2021)
- [Tabular Transformers for Modeling Multivariate Time Series](https://mlsanity.com/paper/2011.01843/) (2020)
- [VAEM: a Deep Generative Model for Heterogeneous Mixed Type Data](https://mlsanity.com/paper/2006.11941/) (2020)
- [Copula Flows for Synthetic Data Generation](https://mlsanity.com/paper/2101.00598/) (2021)
- [Adversarial Generation of Natural Language](https://mlsanity.com/paper/1705.10929/) (2017)

## Built on by (most cited first, 30 of 294)

- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [In-Context Unlearning: Language Models as Few Shot Unlearners](https://mlsanity.com/paper/2310.07579/) (2023)
- [Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space](https://mlsanity.com/paper/2310.09656/) (2023)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [How Can Recommender Systems Benefit from Large Language Models: A Survey](https://mlsanity.com/paper/2306.05817/) (2023)
- [GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoning](https://mlsanity.com/paper/2312.12241/) (2023)
- [Best Practices and Lessons Learned on Synthetic Data](https://mlsanity.com/paper/2404.07503/) (2024)
- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers](https://mlsanity.com/paper/2302.02041/) (2023)
- [ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation](https://mlsanity.com/paper/2308.11131/) (2023)
- [Large Language Models are Competitive Near Cold-start Recommenders for Language- and Item-based Preferences](https://mlsanity.com/paper/2307.14225/) (2023)
- [TabDiff: a Mixed-type Diffusion Model for Tabular Data Generation](https://mlsanity.com/paper/2410.20626/) (2024)
- [Generating and Imputing Tabular Data via Diffusion and Flow-based Gradient-Boosted Trees](https://mlsanity.com/paper/2309.09968/) (2023)
- [Making Pre-trained Language Models Great on Tabular Prediction](https://mlsanity.com/paper/2403.01841/) (2024)
- [Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective](https://mlsanity.com/paper/2406.00793/) (2024)
- [Generative Table Pre-training Empowers Models for Tabular Prediction](https://mlsanity.com/paper/2305.09696/) (2023)
- [ONCE: Boosting Content-based Recommendation with Both Open- and Closed-source Large Language Models](https://mlsanity.com/paper/2305.06566/) (2023)
- [Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes](https://mlsanity.com/paper/2312.12112/) (2023)
- [SoK: Watermarking for AI-Generated Content](https://mlsanity.com/paper/2411.18479/) (2024)
- [TabuLa: Harnessing Language Models for Tabular Data Synthesis](https://mlsanity.com/paper/2310.12746/) (2023)
- [Towards Natural Language-Guided Drones: GeoText-1652 Benchmark with Spatial Relation Matching](https://mlsanity.com/paper/2311.12751/) (2023)
- [TabMT: Generating tabular data with masked transformers](https://mlsanity.com/paper/2312.06089/) (2023)
- [Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models](https://mlsanity.com/paper/2404.06209/) (2024)
- [A Synthetic Dataset for Personal Attribute Inference](https://mlsanity.com/paper/2406.07217/) (2024)
- [Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation](https://mlsanity.com/paper/2408.09698/) (2024)
- [CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks](https://mlsanity.com/paper/2507.23751/) (2025)
- [TABLET: Learning From Instructions For Tabular Data](https://mlsanity.com/paper/2304.13188/) (2023)
- [TabPFGen -- Tabular Data Generation with TabPFN](https://mlsanity.com/paper/2406.05216/) (2024)
- [A Comprehensive Survey of Synthetic Tabular Data Generation](https://mlsanity.com/paper/2504.16506/) (2025)

## Abstract (arXiv)

Tabular data is among the oldest and most ubiquitous forms of data. However, the generation of synthetic samples with the original data's characteristics remains a significant challenge for tabular data. While many generative models from the computer vision domain, such as variational autoencoders or generative adversarial networks, have been adapted for tabular data generation, less research has been directed towards recent transformer-based large language models (LLMs), which are also generative in nature. To this end, we propose GReaT (Generation of Realistic Tabular data), which exploits an auto-regressive generative LLM to sample synthetic and yet highly realistic tabular data. Furthermore, GReaT can model tabular data distributions by conditioning on any subset of features; the remaining features are sampled without additional overhead. We demonstrate the effectiveness of the proposed approach in a series of experiments that quantify the validity and quality of the produced data samples from multiple angles. We find that GReaT maintains state-of-the-art performance across numerous real-world and synthetic data sets with heterogeneous feature types coming in various sizes.