mlsanityData to 7 Oct 2026

Tabby: A Language Model Architecture for Tabular and Structured Data Synthesis

arXiv 2503.02152 ↗First posted 4 Mar 2025cs.LG cs.CLcited by 4

On the map: Synthetic data and self-improvement for LLMs → Privacy-preserving tabular data synthesis

Abstract (arXiv)

While advances in large language models (LLMs) have greatly improved the quality of synthetic text data in recent years, synthesizing tabular data has received relatively less attention. We address this disparity with Tabby, a simple but powerful post-training modification to the standard Transformer language model architecture, enabling its use for tabular dataset synthesis. Tabby enables the representation of differences across columns using Gated Mixture-of-Experts, with column-specific sets of parameters. Empirically, Tabby results in data quality near or equal to that of real data. By pairing our novel LLM table training technique, Plain, with Tabby, we observe up to a 44% improvement in quality over previous methods. We also show that Tabby extends beyond tables to more general structured data, reaching parity with real data on a nested JSON dataset as well.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Privacy-preserving tabular data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 26 of its 51 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

26 of 51 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024HARMONIC: Harnessing LLMs for Tabular Data Synthesis and Privacy Protection6 Aug
  2. 2023TabuLa: Harnessing Language Models for Tabular Data Synthesis19 Oct
  3. 2023Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Space14 Oct

Other references · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  7. 2019Modeling Tabular data using Conditional GANcited by 695
  8. 2022TabDDPM: Modelling Tabular Data with Diffusion Modelscited by 373
  9. 2022Language Models are Realistic Tabular Data Generatorscited by 290
  10. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  11. 2017Equivariance Through Parameter-Sharingcited by 139
  12. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 103
11 more
  1. 2023REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformerscited by 93
  2. 2023Synthcity: facilitating innovative use cases of synthetic data in different data modalitiescited by 76
  3. 2022CTAB-GAN+: Enhancing Tabular Data Synthesiscited by 70
  4. 2023Generating and Imputing Tabular Data via Diffusion and Flow-based Gradient-Boosted Treescited by 60
  5. 2024TabDiff: a Mixed-type Diffusion Model for Tabular Data Generationcited by 60
  6. 2023Generative Table Pre-training Empowers Models for Tabular Predictioncited by 55
  7. 2023Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimescited by 52
  8. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  9. 2024SynthEval: A Framework for Detailed Utility and Privacy Evaluation of Tabular Synthetic Datacited by 27
  10. 2018MEGAN: Mixture of Experts of Generative Adversarial Networks for Multimodal Image Generationcited by 15
  11. 2024Navigating Tabular Data Synthesis Research: Understanding User Needs and Tool Capabilitiescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Research Challenges in Relational Database Management Systems for LLM Queriescited by 2
  2. 2026Team, Then Trim: An Assembly-Line LLM Framework for High-Quality Tabular Data Generationcited by 1
  3. 2026Autoregressive Synthesis of Sparse and Semi-Structured Mixed-Type Datacited by 0
  4. 2026The Power of Order: Fooling LLMs with Adversarial Table Permutationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FASTGEN: Fast and Cost-Effective Synthetic Tabular Data Generation with LLMs
  2. 2023TAP4LLM: Table Provider on Sampling, Augmenting, and Packing Semi-structured Data for Large Language Model Reasoning
  3. 2026Self-Improving Tabular Language Models via Iterative Reward-Guided Post-Training
  4. 2025Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challenges
  5. 2026Why Large Language Models Fail at Tabular Prediction
  6. 2024TabSQLify: Enhancing Reasoning Capabilities of LLMs Through Table Decomposition
  7. 2024Diffusion-nested Auto-Regressive Synthesis of Heterogeneous Tabular Data
  8. 2024EPIC: Effective Prompting for Imbalanced-Class Data Synthesis in Tabular Data Classification via Large Language Models