mlsanityData to 7 Oct 2026

TabTransformer: Tabular Data Modeling Using Contextual Embeddings

arXiv 2012.06678 ↗First posted 11 Dec 2020cs.LG cs.AIcited by 398

Not on the map. Why?

Abstract (arXiv)

We propose TabTransformer, a novel deep tabular data modeling architecture for supervised and semi-supervised learning. The TabTransformer is built upon self-attention based Transformers. The Transformer layers transform the embeddings of categorical features into robust contextual embeddings to achieve higher prediction accuracy. Through extensive experiments on fifteen publicly available datasets, we show that the TabTransformer outperforms the state-of-the-art deep learning methods for tabular data by at least 1.0% on mean AUC, and matches the performance of tree-based ensemble models. Furthermore, we demonstrate that the contextual embeddings learned from TabTransformer are highly robust against both missing and noisy data features, and provide better interpretability. Lastly, for the semi-supervised setting we develop an unsupervised pre-training procedure to learn data-driven contextual embeddings, resulting in an average 2.1% AUC lift over the state-of-the-art methods.

Key facts. Cited by 398 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 27; 2023: 72; 2024: 91; 2025: 106; 2026: 91.

398 papers.

Sort:

By area:

  1. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  2. 2021Tabular Data: Deep Learning is Not All You Needcited by 530
  3. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  4. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 310
  5. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  6. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 218
  7. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 198
  8. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 185
  9. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  10. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  11. 2024OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotationscited by 144
  12. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
38 more
  1. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  2. 2023Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directionscited by 123
  3. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 103
  4. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 97
  5. 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 82
  6. 2023TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023cited by 73
  7. 2023Benchmarking Distribution Shift in Tabular Data with TableShiftcited by 64
  8. 2023Relational Deep Learning: Graph Representation Learning on Relational Databasescited by 55
  9. 2022Transfer Learning with Deep Tabular Modelscited by 52
  10. 2025PyG 2.0: Scalable Learning on Real World Graphscited by 49
  11. 2024Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoningcited by 48
  12. 2024TuneTables: Context Optimization for Scalable Prior-Data Fitted Networkscited by 47
  13. 2024TALENT: A Tabular Analytics and Learning Toolboxcited by 46
  14. 2022Learning to Estimate Shapley Values with Vision Transformerscited by 44
  15. 2024Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extractioncited by 44
  16. 2022Revisiting Pretraining Objectives for Tabular Deep Learningcited by 43
  17. 2022Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Taskscited by 42
  18. 2022T2G-Former: Organizing Tabular Features into Relation Graphs Promotes Heterogeneous Feature Interactioncited by 41
  19. 2025Griffin: Towards a Graph-Centric Relational Database Foundation Modelcited by 41
  20. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 38
  21. 2023Language Models are Few-shot Learners for Prognostic Predictioncited by 37
  22. 2024SwitchTab: Switched Autoencoders Are Effective Tabular Learnerscited by 37
  23. 2023Trompt: Towards a Better Deep Neural Network for Tabular Datacited by 36
  24. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  25. 2023HYTREL: Hypergraph-enhanced Tabular Data Representation Learningcited by 35
  26. 2023ReMasker: Imputing Tabular Data with Masked Autoencodingcited by 35
  27. 2024OmniPred: Language Models as Universal Regressorscited by 34
  28. 2025TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problemscited by 33
  29. 2024PyTorch Frame: A Modular Framework for Multi-Modal Tabular Learningcited by 32
  30. 2024Mixture of In-Context Prompters for Tabular PFNscited by 32
  31. 2024TIP: Tabular-Image Pre-training for Multimodal Classification with Incomplete Datacited by 32
  32. 2022PTab: Using the Pre-trained Language Model for Modeling Tabular Datacited by 30
  33. 2023TABLET: Learning From Instructions For Tabular Datacited by 28
  34. 2023UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Sciencecited by 28
  35. 2024PTaRL: Prototype-based Tabular Representation Learning via Space Calibrationcited by 27
  36. 2021Benchmarking Multimodal AutoML for Tabular Data with Text Fieldscited by 24
  37. 2023FlowTransformer: A Transformer Framework for Flow-based Network Intrusion Detection Systemscited by 24
  38. 2023MediTab: Scaling Medical Tabular Data Predictors via Data Consolidation, Enrichment, and Refinementcited by 24

50 most cited of 398 All 398 (JSON)