# SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning

arXiv: https://arxiv.org/abs/2110.04361 · HTML: https://mlsanity.com/paper/2110.04361/

## Key facts

- Cited by 104 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 48 references that are in the map.
- First posted on arXiv 8 Oct 2021.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Invariant Information Clustering for Unsupervised Image Classification and Segmentation](https://mlsanity.com/paper/1807.06653/) (2018)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [Adversarial Autoencoders](https://mlsanity.com/paper/1511.05644/) (2015)
- [OpenML Benchmarking Suites](https://mlsanity.com/paper/1708.03731/) (2017)
- [Learning Discrete Representations via Information Maximizing Self-Augmented Training](https://mlsanity.com/paper/1702.08720/) (2017)
- [Stacked Capsule Autoencoders](https://mlsanity.com/paper/1906.06818/) (2019)
- [A Framework For Contrastive Self-Supervised Learning And Designing A New Approach](https://mlsanity.com/paper/2009.00104/) (2020)
- [Improving Self-Organizing Maps with Unsupervised Feature Extraction](https://mlsanity.com/paper/2009.02174/) (2020)

## Built on by (most cited first, 30 of 104)

- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [XTab: Cross-table Pretraining for Tabular Transformers](https://mlsanity.com/paper/2305.06090/) (2023)
- [Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning](https://mlsanity.com/paper/2404.09491/) (2024)
- [To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Review](https://mlsanity.com/paper/2304.09355/) (2023)
- [Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data](https://mlsanity.com/paper/2303.14080/) (2023)
- [Transfer Learning with Deep Tabular Models](https://mlsanity.com/paper/2206.15306/) (2022)
- [Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach](https://mlsanity.com/paper/2405.15613/) (2024)
- [STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables](https://mlsanity.com/paper/2303.00918/) (2023)
- [Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning](https://mlsanity.com/paper/2406.08527/) (2024)
- [Revisiting Pretraining Objectives for Tabular Deep Learning](https://mlsanity.com/paper/2207.03208/) (2022)
- [A Comprehensive Survey on Data Augmentation](https://mlsanity.com/paper/2405.09591/) (2024)
- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specialization](https://mlsanity.com/paper/2303.05506/) (2023)
- [TIP: Tabular-Image Pre-training for Multimodal Classification with Incomplete Data](https://mlsanity.com/paper/2407.07582/) (2024)
- [PTab: Using the Pre-trained Language Model for Modeling Tabular Data](https://mlsanity.com/paper/2209.08060/) (2022)
- [Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Later](https://mlsanity.com/paper/2407.03257/) (2024)
- [Novel Class Discovery: an Introduction and Key Concepts](https://mlsanity.com/paper/2302.12028/) (2023)
- [MediTab: Scaling Medical Tabular Data Predictors via Data Consolidation, Enrichment, and Refinement](https://mlsanity.com/paper/2305.12081/) (2023)
- [From Supervised to Generative: A Novel Paradigm for Tabular Deep Learning with Large Language Models](https://mlsanity.com/paper/2310.07338/) (2023)
- [A Performance-Driven Benchmark for Feature Selection in Tabular Deep Learning](https://mlsanity.com/paper/2311.05877/) (2023)
- [Towards Cross-Table Masked Pretraining for Web Data Mining](https://mlsanity.com/paper/2307.04308/) (2023)
- [Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolution](https://mlsanity.com/paper/2408.10548/) (2024)
- [A Survey on Deep Tabular Learning](https://mlsanity.com/paper/2410.12034/) (2024)
- [Self-supervised Representation Learning From Random Data Projectors](https://mlsanity.com/paper/2310.07756/) (2023)
- [A Survey on Self-Supervised Learning for Non-Sequential Tabular Data](https://mlsanity.com/paper/2402.01204/) (2024)
- [CTSyn: A Foundation Model for Cross Tabular Data Generation](https://mlsanity.com/paper/2406.04619/) (2024)
- [T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Data](https://mlsanity.com/paper/2410.05016/) (2024)
- [Graph Neural Networks for Tabular Data Learning: A Survey with Taxonomy and Directions](https://mlsanity.com/paper/2401.02143/) (2024)
- [Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domains](https://mlsanity.com/paper/2405.07414/) (2024)

## Abstract (arXiv)

Self-supervised learning has been shown to be very effective in learning useful representations, and yet much of the success is achieved in data types such as images, audio, and text. The success is mainly enabled by taking advantage of spatial, temporal, or semantic structure in the data through augmentation. However, such structure may not exist in tabular datasets commonly used in fields such as healthcare, making it difficult to design an effective augmentation method, and hindering a similar progress in tabular data setting. In this paper, we introduce a new framework, Subsetting features of Tabular data (SubTab), that turns the task of learning from tabular data into a multi-view representation learning problem by dividing the input features to multiple subsets. We argue that reconstructing the data from the subset of its features rather than its corrupted version in an autoencoder setting can better capture its underlying latent representation. In this framework, the joint representation can be expressed as the aggregate of latent variables of the subsets at test time, which we refer to as collaborative inference. Our experiments show that the SubTab achieves the state of the art (SOTA) performance of 98.31% on MNIST in tabular setting, on par with CNN-based SOTA models, and surpasses existing baselines on three other real-world datasets by a significant margin.