# Rethinking Data Augmentation for Tabular Data in Deep Learning

arXiv: https://arxiv.org/abs/2305.10308 · HTML: https://mlsanity.com/paper/2305.10308/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 53 references that are in the map.
- First posted on arXiv 17 May 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [Synthetic Data from Diffusion Models Improves ImageNet Classification](https://mlsanity.com/paper/2304.08466/) (2023)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [Effective Data Augmentation With Diffusion Models](https://mlsanity.com/paper/2302.07944/) (2023)
- [Data Augmentation Generative Adversarial Networks](https://mlsanity.com/paper/1711.04340/) (2017)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption](https://mlsanity.com/paper/2106.15147/) (2021)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning](https://mlsanity.com/paper/2110.04361/) (2021)
- [BAGAN: Data Augmentation with Balancing GAN](https://mlsanity.com/paper/1803.09655/) (2018)
- [Conditional Wasserstein GAN-based Oversampling of Tabular Data for Imbalanced Learning](https://mlsanity.com/paper/2008.09202/) (2020)
- [Diffusion-based Data Augmentation for Skin Disease Classification: Impact Across Original Medical Datasets to Fully Synthetic Images](https://mlsanity.com/paper/2301.04802/) (2023)
- [ExcelFormer: A neural network surpassing GBDTs on tabular data](https://mlsanity.com/paper/2301.02819/) (2023)
- [Contrastive Mixup: Self- and Semi-Supervised learning for Tabular Domain](https://mlsanity.com/paper/2108.12296/) (2021)
- [TabRet: Pre-training Transformer-based Tabular Models for Unseen Columns](https://mlsanity.com/paper/2303.15747/) (2023)
- [Fruit Quality and Defect Image Classification with Conditional GAN Data Augmentation](https://mlsanity.com/paper/2104.05647/) (2021)
- [Multimodal Data Augmentation for Image Captioning using Diffusion Models](https://mlsanity.com/paper/2305.01855/) (2023)

## Built on by (most cited first, 10 of 10)

- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [TimeDRL: Disentangled Representation Learning for Multivariate Time-Series](https://mlsanity.com/paper/2312.04142/) (2023)
- [TabRep: Training Tabular Diffusion Models with a Simple and Effective Continuous Representation](https://mlsanity.com/paper/2504.04798/) (2025)
- [Fusing Echocardiography Images and Medical Records for Continuous Patient Stratification](https://mlsanity.com/paper/2401.07796/) (2024)
- [Self-Supervised Learning of Disentangled Representations for Multivariate Time-Series](https://mlsanity.com/paper/2410.12606/) (2024)
- [Active Tabular Augmentation via Policy-Guided Diffusion Inpainting](https://mlsanity.com/paper/2605.10315/) (2026)
- [Anomaly Detection in Power Generation Plants with Generative Adversarial Networks](https://mlsanity.com/paper/2310.00335/) (2023)
- [Self-supervised learning of multi-omics embeddings in the low-label, high-data regime](https://mlsanity.com/paper/2311.09962/) (2023)
- [SAFLEX: Self-Adaptive Augmentation via Feature Label Extrapolation](https://mlsanity.com/paper/2410.02512/) (2024)
- [A Systematic Review of Digital Twin-Driven Predictive Maintenance in Industrial Engineering: Taxonomy, Architectural Elements, and Future Research Directions](https://mlsanity.com/paper/2509.24443/) (2025)

## Abstract (arXiv)

Tabular data is the most widely used data format in machine learning (ML). While tree-based methods outperform DL-based methods in supervised learning, recent literature reports that self-supervised learning with Transformer-based models outperforms tree-based methods. In the existing literature on self-supervised learning for tabular data, contrastive learning is the predominant method. In contrastive learning, data augmentation is important to generate different views. However, data augmentation for tabular data has been difficult due to the unique structure and high complexity of tabular data. In addition, three main components are proposed together in existing methods: model structure, self-supervised learning methods, and data augmentation. Therefore, previous works have compared the performance without comprehensively considering these components, and it is not clear how each component affects the actual performance. In this study, we focus on data augmentation to address these issues. We propose a novel data augmentation method, Mask Token Replacement (textttMTR), which replaces the mask token with a portion of each tokenized column; textttMTR takes advantage of the properties of Transformer, which is becoming the predominant DL-based architecture for tabular data, to perform data augmentation for each column embedding. Through experiments with 13 diverse public datasets in both supervised and self-supervised learning scenarios, we show that textttMTR achieves competitive performance against existing data augmentation methods and improves model performance. In addition, we discuss specific scenarios in which textttMTR is most effective and identify the scope of its application. The code is available at https://github.com/somaonishi/MTR/.