# TabTransformer: Tabular Data Modeling Using Contextual Embeddings

arXiv: https://arxiv.org/abs/2012.06678 · HTML: https://mlsanity.com/paper/2012.06678/

## Key facts

- Cited by 409 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2020.

## Built on by (most cited first, 30 of 409)

- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations](https://mlsanity.com/paper/2412.07626/) (2024)
- [LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks](https://mlsanity.com/paper/2206.06565/) (2022)
- [Meta-Transformer: A Unified Framework for Multimodal Learning](https://mlsanity.com/paper/2307.10802/) (2023)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions](https://mlsanity.com/paper/2310.19775/) (2023)
- [TabDPT: Scaling Tabular Foundation Models on Real Data](https://mlsanity.com/paper/2410.18164/) (2024)
- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [Reliability of CKA as a Similarity Measure in Deep Learning](https://mlsanity.com/paper/2210.16156/) (2022)
- [TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023](https://mlsanity.com/paper/2307.14338/) (2023)
- [Benchmarking Distribution Shift in Tabular Data with TableShift](https://mlsanity.com/paper/2312.07577/) (2023)
- [Relational Deep Learning: Graph Representation Learning on Relational Databases](https://mlsanity.com/paper/2312.04615/) (2023)
- [Transfer Learning with Deep Tabular Models](https://mlsanity.com/paper/2206.15306/) (2022)
- [TALENT: A Tabular Analytics and Learning Toolbox](https://mlsanity.com/paper/2407.04057/) (2024)
- [PyG 2.0: Scalable Learning on Real World Graphs](https://mlsanity.com/paper/2507.16991/) (2025)
- [Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning](https://mlsanity.com/paper/2406.08527/) (2024)
- [TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks](https://mlsanity.com/paper/2402.11137/) (2024)
- [Griffin: Towards a Graph-Centric Relational Database Foundation Model](https://mlsanity.com/paper/2505.05568/) (2025)
- [Learning to Estimate Shapley Values with Vision Transformers](https://mlsanity.com/paper/2206.05282/) (2022)
- [Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction](https://mlsanity.com/paper/2410.21169/) (2024)
- [Revisiting Pretraining Objectives for Tabular Deep Learning](https://mlsanity.com/paper/2207.03208/) (2022)
- [Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks](https://mlsanity.com/paper/2201.09745/) (2022)

## Abstract (arXiv)

We propose TabTransformer, a novel deep tabular data modeling architecture for supervised and semi-supervised learning. The TabTransformer is built upon self-attention based Transformers. The Transformer layers transform the embeddings of categorical features into robust contextual embeddings to achieve higher prediction accuracy. Through extensive experiments on fifteen publicly available datasets, we show that the TabTransformer outperforms the state-of-the-art deep learning methods for tabular data by at least 1.0% on mean AUC, and matches the performance of tree-based ensemble models. Furthermore, we demonstrate that the contextual embeddings learned from TabTransformer are highly robust against both missing and noisy data features, and provide better interpretability. Lastly, for the semi-supervised setting we develop an unsupervised pre-training procedure to learn data-driven contextual embeddings, resulting in an average 2.1% AUC lift over the state-of-the-art methods.