# TabMT: Generating tabular data with masked transformers

arXiv: https://arxiv.org/abs/2312.06089 · HTML: https://mlsanity.com/paper/2312.06089/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 11 Dec 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Improved Precision and Recall Metric for Assessing Generative Models](https://mlsanity.com/paper/1904.06991/) (2019)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Modeling Tabular data using Conditional GAN](https://mlsanity.com/paper/1907.00503/) (2019)
- [TabDDPM: Modelling Tabular Data with Diffusion Models](https://mlsanity.com/paper/2209.15421/) (2022)
- [Language Models are Realistic Tabular Data Generators](https://mlsanity.com/paper/2210.06280/) (2022)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [CatBoost: gradient boosting with categorical features support](https://mlsanity.com/paper/1810.11363/) (2018)
- [Understanding How Dimension Reduction Tools Work: An Empirical Approach to Deciphering t-SNE, UMAP, TriMAP, and PaCMAP for Data Visualization](https://mlsanity.com/paper/2012.04456/) (2020)
- [TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up](https://mlsanity.com/paper/2102.07074/) (2021)
- [CTAB-GAN: Effective Table Data Synthesizing](https://mlsanity.com/paper/2102.08369/) (2021)
- [REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers](https://mlsanity.com/paper/2302.02041/) (2023)
- [STaSy: Score-based Tabular data Synthesis](https://mlsanity.com/paper/2210.04018/) (2022)
- [Tabular Transformers for Modeling Multivariate Time Series](https://mlsanity.com/paper/2011.01843/) (2020)
- [CTAB-GAN+: Enhancing Tabular Data Synthesis](https://mlsanity.com/paper/2204.00401/) (2022)
- [Conditional Wasserstein GAN-based Oversampling of Tabular Data for Imbalanced Learning](https://mlsanity.com/paper/2008.09202/) (2020)
- [Flow-based Network Traffic Generation using Generative Adversarial Networks](https://mlsanity.com/paper/1810.07795/) (2018)

## Built on by (most cited first, 30 of 36)

- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [A Comprehensive Survey of Synthetic Tabular Data Generation](https://mlsanity.com/paper/2504.16506/) (2025)
- [Large Language Models are Good Relational Learners](https://mlsanity.com/paper/2506.05725/) (2025)
- [Differentially Private Tabular Data Synthesis using Large Language Models](https://mlsanity.com/paper/2406.01457/) (2024)
- [TabularARGN: A Flexible and Efficient Auto-Regressive Framework for Generating High-Fidelity Synthetic Data](https://mlsanity.com/paper/2501.12012/) (2025)
- [TabGen-ICL: Residual-Aware In-Context Example Selection for Tabular Data Generation](https://mlsanity.com/paper/2502.16414/) (2025)
- [Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarking](https://mlsanity.com/paper/2401.04266/) (2024)
- [Generating Realistic Tabular Data with Large Language Models](https://mlsanity.com/paper/2410.21717/) (2024)
- [Causality for Tabular Data Synthesis: A High-Order Structure Causal Benchmark Framework](https://mlsanity.com/paper/2406.08311/) (2024)
- [Synthetic Tabular Data Generation for Imbalanced Classification: The Surprising Effectiveness of an Overlap Class](https://mlsanity.com/paper/2412.15657/) (2024)
- [Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Models](https://mlsanity.com/paper/2507.18504/) (2025)
- [MALLM-GAN: Multi-Agent Large Language Model as Generative Adversarial Network for Synthesizing Tabular Data](https://mlsanity.com/paper/2406.10521/) (2024)
- [Deep Feature Embedding for Tabular Data](https://mlsanity.com/paper/2408.17162/) (2024)
- [A Context-Aware Approach for Enhancing Data Imputation with Pre-trained Language Models](https://mlsanity.com/paper/2405.17712/) (2024)
- [Diffusion Transformers for Tabular Data Time Series Generation](https://mlsanity.com/paper/2504.07566/) (2025)
- [Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation](https://mlsanity.com/paper/2510.09671/) (2025)
- [Unmasking Trees for Tabular Data](https://mlsanity.com/paper/2407.05593/) (2024)
- [Diffusion-nested Auto-Regressive Synthesis of Heterogeneous Tabular Data](https://mlsanity.com/paper/2410.21523/) (2024)
- [iTBLS: A Dataset of Interactive Conversations Over Tabular Information](https://mlsanity.com/paper/2404.12580/) (2024)
- [MaskSDM with Shapley values to improve flexibility, robustness, and explainability in species distribution modeling](https://mlsanity.com/paper/2503.13057/) (2025)
- [Graph-Conditional Flow Matching for Relational Data Generation](https://mlsanity.com/paper/2505.15668/) (2025)
- [TabTreeFormer: Tabular Data Generation Using Hybrid Tree-Transformer](https://mlsanity.com/paper/2501.01216/) (2025)
- [A Generalized Theory of Mixup for Structure-Preserving Synthetic Data](https://mlsanity.com/paper/2503.02645/) (2025)
- [Masked Language Modeling Becomes Conditional Density Estimation for Tabular Data Synthesis](https://mlsanity.com/paper/2405.20602/) (2024)
- [TAEGAN: Generating Synthetic Tabular Data For Data Augmentation](https://mlsanity.com/paper/2410.01933/) (2024)
- [NRGBoost: Energy-Based Generative Boosted Trees](https://mlsanity.com/paper/2410.03535/) (2024)
- [MUSE: Model-Agnostic Tabular Watermarking via Multi-Sample Selection](https://mlsanity.com/paper/2505.24267/) (2025)
- [Democratizing Tabular Data Access with an Openunicodex2013Source Syntheticunicodex2013Data SDK](https://mlsanity.com/paper/2508.00718/) (2025)
- [Towards Universal Debiasing for Language Models-based Tabular Data Generation](https://mlsanity.com/paper/2509.16475/) (2025)
- [Robust Spectral Watermark for Synthetic Tabular Data](https://mlsanity.com/paper/2511.21600/) (2025)

## Abstract (arXiv)

Autoregressive and Masked Transformers are incredibly effective as generative models and classifiers. While these models are most prevalent in NLP, they also exhibit strong performance in other domains, such as vision. This work contributes to the exploration of transformer-based models in synthetic data generation for diverse application domains. In this paper, we present TabMT, a novel Masked Transformer design for generating synthetic tabular data. TabMT effectively addresses the unique challenges posed by heterogeneous data fields and is natively able to handle missing data. Our design leverages improved masking techniques to allow for generation and demonstrates state-of-the-art performance from extremely small to extremely large tabular datasets. We evaluate TabMT for privacy-focused applications and find that it is able to generate high quality data with superior privacy tradeoffs.