# Tabby: A Language Model Architecture for Tabular and Structured Data Synthesis

arXiv: https://arxiv.org/abs/2503.02152 · HTML: https://mlsanity.com/paper/2503.02152/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Privacy-preserving tabular data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 26 of its 51 references that are in the map.
- First posted on arXiv 4 Mar 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [Privacy-preserving tabular data synthesis](https://mlsanity.com/synth/privacy-preserving-tabular-data-synthesis/)

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [TabDDPM: Modelling Tabular Data with Diffusion Models](https://mlsanity.com/paper/2209.15421/) (2022)
- [Language Models are Realistic Tabular Data Generators](https://mlsanity.com/paper/2210.06280/) (2022)
- [Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space](https://mlsanity.com/paper/2310.09656/) (2023)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [Equivariance Through Parameter-Sharing](https://mlsanity.com/paper/1702.08389/) (2017)
- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers](https://mlsanity.com/paper/2302.02041/) (2023)
- [Synthcity: facilitating innovative use cases of synthetic data in different data modalities](https://mlsanity.com/paper/2301.07573/) (2023)
- [CTAB-GAN+: Enhancing Tabular Data Synthesis](https://mlsanity.com/paper/2204.00401/) (2022)
- [TabDiff: a Mixed-type Diffusion Model for Tabular Data Generation](https://mlsanity.com/paper/2410.20626/) (2024)
- [Generating and Imputing Tabular Data via Diffusion and Flow-based Gradient-Boosted Trees](https://mlsanity.com/paper/2309.09968/) (2023)
- [Generative Table Pre-training Empowers Models for Tabular Prediction](https://mlsanity.com/paper/2305.09696/) (2023)
- [Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimes](https://mlsanity.com/paper/2312.12112/) (2023)
- [Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners](https://mlsanity.com/paper/2204.07689/) (2022)
- [TabuLa: Harnessing Language Models for Tabular Data Synthesis](https://mlsanity.com/paper/2310.12746/) (2023)
- [SynthEval: A Framework for Detailed Utility and Privacy Evaluation of Tabular Synthetic Data](https://mlsanity.com/paper/2404.15821/) (2024)
- [HARMONIC: Harnessing LLMs for Tabular Data Synthesis and Privacy Protection](https://mlsanity.com/paper/2408.02927/) (2024)
- [MEGAN: Mixture of Experts of Generative Adversarial Networks for Multimodal Image Generation](https://mlsanity.com/paper/1805.02481/) (2018)
- [Navigating Tabular Data Synthesis Research: Understanding User Needs and Tool Capabilities](https://mlsanity.com/paper/2405.20959/) (2024)

## Built on by (most cited first, 4 of 4)

- [Research Challenges in Relational Database Management Systems for LLM Queries](https://mlsanity.com/paper/2508.20912/) (2025)
- [Team, Then Trim: An Assembly-Line LLM Framework for High-Quality Tabular Data Generation](https://mlsanity.com/paper/2602.04785/) (2026)
- [The Power of Order: Fooling LLMs with Adversarial Table Permutations](https://mlsanity.com/paper/2605.00445/) (2026)
- [Autoregressive Synthesis of Sparse and Semi-Structured Mixed-Type Data](https://mlsanity.com/paper/2603.01444/) (2026)

## Abstract (arXiv)

While advances in large language models (LLMs) have greatly improved the quality of synthetic text data in recent years, synthesizing tabular data has received relatively less attention. We address this disparity with Tabby, a simple but powerful post-training modification to the standard Transformer language model architecture, enabling its use for tabular dataset synthesis. Tabby enables the representation of differences across columns using Gated Mixture-of-Experts, with column-specific sets of parameters. Empirically, Tabby results in data quality near or equal to that of real data. By pairing our novel LLM table training technique, Plain, with Tabby, we observe up to a 44% improvement in quality over previous methods. We also show that Tabby extends beyond tables to more general structured data, reaching parity with real data on a nested JSON dataset as well.