# Contrastive Mixup: Self- and Semi-Supervised learning for Tabular Domain

arXiv: https://arxiv.org/abs/2108.12296 · HTML: https://mlsanity.com/paper/2108.12296/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 27 Aug 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning](https://mlsanity.com/paper/1606.04586/) (2016)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [Label Propagation for Deep Semi-supervised Learning](https://mlsanity.com/paper/1904.04717/) (2019)
- [An efficient framework for learning sentence representations](https://mlsanity.com/paper/1803.02893/) (2018)
- [Low-shot learning with large-scale diffusion](https://mlsanity.com/paper/1706.02332/) (2017)

## Built on by (most cited first, 20 of 20)

- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [Revisiting Pretraining Objectives for Tabular Deep Learning](https://mlsanity.com/paper/2207.03208/) (2022)
- [A Comprehensive Survey on Data Augmentation](https://mlsanity.com/paper/2405.09591/) (2024)
- [ExcelFormer: A neural network surpassing GBDTs on tabular data](https://mlsanity.com/paper/2301.02819/) (2023)
- [Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolution](https://mlsanity.com/paper/2408.10548/) (2024)
- [Hopular: Modern Hopfield Networks for Tabular Data](https://mlsanity.com/paper/2206.00664/) (2022)
- [A Survey on Deep Tabular Learning](https://mlsanity.com/paper/2410.12034/) (2024)
- [HyperTab: Hypernetwork Approach for Deep Learning on Small Tabular Datasets](https://mlsanity.com/paper/2304.03543/) (2023)
- [Rethinking Data Augmentation for Tabular Data in Deep Learning](https://mlsanity.com/paper/2305.10308/) (2023)
- [Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarking](https://mlsanity.com/paper/2401.04266/) (2024)
- [Local Contrastive Feature learning for Tabular Data](https://mlsanity.com/paper/2211.10549/) (2022)
- [Scaling Experiments in Self-Supervised Cross-Table Representation Learning](https://mlsanity.com/paper/2309.17339/) (2023)
- [Tree-Regularized Tabular Embeddings](https://mlsanity.com/paper/2403.00963/) (2024)
- [Revisiting Self-Training with Regularized Pseudo-Labeling for Tabular Data](https://mlsanity.com/paper/2302.14013/) (2023)
- [Boarding for ISS: Imbalanced Self-Supervised: Discovery of a Scaled Autoencoder for Mixed Tabular Datasets](https://mlsanity.com/paper/2403.15790/) (2024)
- [Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcity](https://mlsanity.com/paper/2308.00177/) (2023)
- [Active Tabular Augmentation via Policy-Guided Diffusion Inpainting](https://mlsanity.com/paper/2605.10315/) (2026)
- [Generalized Supervised Contrastive Learning](https://mlsanity.com/paper/2206.00384/) (2022)
- [Progressive Feature Upgrade in Semi-supervised Learning on Tabular Domain](https://mlsanity.com/paper/2212.00892/) (2022)
- [Q-Match: Self-Supervised Learning by Matching Distributions Induced by a Queue](https://mlsanity.com/paper/2302.05444/) (2023)

## Abstract (arXiv)

Recent literature in self-supervised has demonstrated significant progress in closing the gap between supervised and unsupervised methods in the image and text domains. These methods rely on domain-specific augmentations that are not directly amenable to the tabular domain. Instead, we introduce Contrastive Mixup, a semi-supervised learning framework for tabular data and demonstrate its effectiveness in limited annotated data settings. Our proposed method leverages Mixup-based augmentation under the manifold assumption by mapping samples to a low dimensional latent space and encourage interpolated samples to have high a similarity within the same labeled class. Unlabeled samples are additionally employed via a transductive label propagation method to further enrich the set of similar and dissimilar pairs that can be used in the contrastive loss term. We demonstrate the effectiveness of the proposed framework on public tabular datasets and real-world clinical datasets.