mlsanityData to 7 Oct 2026

Tabular Data: Deep Learning is Not All You Need

arXiv 2106.03253 ↗First posted 6 Jun 2021cs.LGcited by 530

Not on the map. Why?

Abstract (arXiv)

A key element in solving real-life data science problems is selecting the types of models to use. Tree ensemble models (such as XGBoost) are usually recommended for classification and regression problems with tabular data. However, several deep learning models for tabular data have recently been proposed, claiming to outperform XGBoost for some use cases. This paper explores whether these deep models should be a recommended option for tabular data by rigorously comparing the new deep models to XGBoost on various datasets. In addition to systematically comparing their performance, we consider the tuning and computation they require. Our study shows that XGBoost outperforms these deep models across the datasets, including the datasets used in the papers that proposed the deep models. We also demonstrate that XGBoost requires much less tuning. On the positive side, we show that an ensemble of deep models and XGBoost performs better on these datasets than XGBoost alone.

Key facts. Cited by 530 papers in the map (data through 7 Oct 2026). Builds on 21 of its 58 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

21 of 58 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  7. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  8. 2017Self-Normalizing Neural Networkscited by 591
  9. 2014OpenML: networked science in machine learningcited by 523
  10. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  11. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 398
  12. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 310
9 more
  1. 2018CatBoost: gradient boosting with categorical features supportcited by 208
  2. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 200
  3. 2013Introducing LETOR 4.0 Datasetscited by 120
  4. 2019Theoretical Issues in Deep Networks: Approximation, Optimization and Generalizationcited by 72
  5. 2020The Tree Ensemble Layer: Differentiability meets Conditional Computationcited by 57
  6. 2020Gradient Boosting Neural Networks: GrowNetcited by 46
  7. 2020DNF-Net: A Neural Architecture for Tabular Datacited by 17
  8. 2020The Dual Information Bottleneckcited by 14
  9. 2018XGBoost: Scalable GPU Accelerated Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 55; 2023: 116; 2024: 120; 2025: 122; 2026: 110.

530 papers.

Sort:

By area:

  1. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  2. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  3. 2023A Comprehensive Survey on Test-Time Adaptation under Distribution Shiftscited by 353
  4. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  5. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 218
  6. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 198
  7. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 185
  8. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  9. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  10. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  11. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 103
  12. 2023REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformerscited by 93
38 more
  1. 2023CoDi: Co-evolving Contrastive Diffusion Models for Mixed-type Tabular Synthesiscited by 78
  2. 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 73
  3. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  4. 2024CARTE: Pretraining and Transfer for Tabular Learningcited by 69
  5. 2023Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Datacited by 68
  6. 2023Benchmarking Distribution Shift in Tabular Data with TableShiftcited by 64
  7. 2023Generating and Imputing Tabular Data via Diffusion and Flow-based Gradient-Boosted Treescited by 60
  8. 2024Retrieval & Fine-Tuning for In-Context Tabular Modelscited by 58
  9. 2023Generative Table Pre-training Empowers Models for Tabular Predictioncited by 55
  10. 2023Relational Deep Learning: Graph Representation Learning on Relational Databasescited by 55
  11. 2022Transfer Learning with Deep Tabular Modelscited by 52
  12. 2023Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimescited by 52
  13. 2023STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tablescited by 48
  14. 2024TuneTables: Context Optimization for Scalable Prior-Data Fitted Networkscited by 47
  15. 2022Revisiting Pretraining Objectives for Tabular Deep Learningcited by 43
  16. 2023Trompt: Towards a Better Deep Neural Network for Tabular Datacited by 36
  17. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  18. 2021Locally Sparse Neural Networks for Tabular Biomedical Datacited by 34
  19. 2023TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specializationcited by 32
  20. 2024PyTorch Frame: A Modular Framework for Multi-Modal Tabular Learningcited by 32
  21. 2024Mixture of In-Context Prompters for Tabular PFNscited by 32
  22. 2025Relational Transformer: Toward Zero-Shot Foundation Models for Relational Datacited by 32
  23. 2023Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Datacited by 31
  24. 2022PTab: Using the Pre-trained Language Model for Modeling Tabular Datacited by 30
  25. 2024Review of multimodal machine learning approaches in healthcarecited by 30
  26. 2023Beyond Privacy: Navigating the Opportunities and Challenges of Synthetic Datacited by 29
  27. 2023Synthetic data, real errors: how (not) to publish and use synthetic datacited by 29
  28. 2024TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarkscited by 29
  29. 2023TABLET: Learning From Instructions For Tabular Datacited by 28
  30. 2024PTaRL: Prototype-based Tabular Representation Learning via Space Calibrationcited by 27
  31. 2022FairGBM: Gradient Boosting with Fairness Constraintscited by 25
  32. 2024Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Datacited by 25
  33. 2022Benchmarking missing-values approaches for predictive models on health databasescited by 23
  34. 2023Reimagining Synthetic Tabular Data Generation through Data-Centric AI: A Comprehensive Benchmarkcited by 23
  35. 2021Counterfactual Shapley Additive Explanationscited by 22
  36. 2022Data-IQ: Characterizing subgroups with heterogeneous outcomes in tabular datacited by 22
  37. 2024HyperFast: Instant Classification for Tabular Datacited by 22
  38. 2023Language models are weak learnerscited by 21

50 most cited of 530 All 530 (JSON)