mlsanityData to 14 Oct 2026

Revisiting Deep Learning Models for Tabular Data

arXiv 2106.11959 ↗First posted 22 Jun 2021cs.LGcited by 740

Not on the map. Why?

Cited by 20 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The existing literature on deep learning for tabular data proposes a wide range of novel architectures and reports competitive results on various datasets. However, the proposed models are usually not properly compared to each other and existing works often use different benchmarks and experiment protocols. As a result, it is unclear for both researchers and practitioners what models perform best. Additionally, the field still lacks effective baselines, that is, the easy-to-use models that provide competitive performance across different problems. In this work, we perform an overview of the main families of DL architectures for tabular data and raise the bar of baselines in tabular DL by identifying two simple and powerful deep architectures. The first one is a ResNet-like architecture which turns out to be a strong baseline that is often missing in prior works. The second model is our simple adaptation of the Transformer architecture for tabular data, which outperforms other solutions on most tasks. Both models are compared to many existing architectures on a diverse set of tasks under the same training and tuning protocols. We also compare the best DL models with Gradient Boosted Decision Trees and conclude that there is still no universally superior solution.

Key facts. Cited by 740 papers in the map (data through 14 Oct 2026). Builds on 37 of its 63 references that are in the map. First posted on arXiv 22 Jun 2021.

Builds on

37 of 63 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2016Layer Normalizationcited by 5,669
  8. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  11. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  12. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
25 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  2. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  5. 2020Efficient Transformers: A Surveycited by 790
  6. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  7. 2017Self-Normalizing Neural Networkscited by 598
  8. 2014OpenML: networked science in machine learningcited by 526
  9. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
  10. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  11. 2017Deep & Cross Network for Ad Click Predictionscited by 393
  12. 2020DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systemscited by 327
  13. 2018AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networkscited by 320
  14. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  15. 2020Understanding the Difficulty of Training Transformerscited by 210
  16. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 203
  17. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  18. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 148
  19. 2013Introducing LETOR 4.0 Datasetscited by 121
  20. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  21. 2018Deep Neural Decision Treescited by 72
  22. 2020The Tree Ensemble Layer: Differentiability meets Conditional Computationcited by 57
  23. 2020Gradient Boosting Neural Networks: GrowNetcited by 46
  24. 2021Which transformer architecture fits my data? A vocabulary bottleneck in self-attentioncited by 16
  25. 2021Revisiting Simple Neural Probabilistic Language Modelscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 50; 2023: 122; 2024: 143; 2025: 190; 2026: 228.

740 papers.

Sort:

By area:

  1. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
  2. 2022TabDDPM: Modelling Tabular Data with Diffusion Modelscited by 377
  3. 2022ADBench: Anomaly Detection Benchmarkcited by 232
  4. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
  5. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 199
  6. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 186
  7. 2023Mixed-Type Tabular Data Synthesis with Score-based Diffusion in Latent Spacecited by 186
  8. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 159
  9. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  10. 2023Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directionscited by 125
  11. 2024TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensemblingcited by 113
  12. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 106
38 more
  1. 2021Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Taskscited by 101
  2. 2024RelBench: A Benchmark for Deep Learning on Relational Databasescited by 96
  3. 2023REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformerscited by 94
  4. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  5. 2023CoDi: Co-evolving Contrastive Diffusion Models for Mixed-type Tabular Synthesiscited by 78
  6. 2022Diffusion models for missing value imputation in tabular datacited by 77
  7. 2025Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Modelscited by 77
  8. 2024CARTE: Pretraining and Transfer for Tabular Learningcited by 76
  9. 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 76
  10. 2023TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023cited by 74
  11. 2023On Diffusion Modeling for Anomaly Detectioncited by 72
  12. 2023Benchmarking Distribution Shift in Tabular Data with TableShiftcited by 66
  13. 2021YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimizationcited by 62
  14. 2024Making Pre-trained Language Models Great on Tabular Predictioncited by 57
  15. 2023Relational Deep Learning: Graph Representation Learning on Relational Databasescited by 56
  16. 2023Generative Table Pre-training Empowers Models for Tabular Predictioncited by 55
  17. 2022Transfer Learning with Deep Tabular Modelscited by 52
  18. 2022OpenFE: Automated Feature Generation with Expert-level Performancecited by 52
  19. 2024TALENT: A Tabular Analytics and Learning Toolboxcited by 52
  20. 2025PyG 2.0: Scalable Learning on Real World Graphscited by 52
  21. 2022A Framework and Benchmark for Deep Batch Active Learning for Regressioncited by 49
  22. 2024Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoningcited by 49
  23. 2024TuneTables: Context Optimization for Scalable Prior-Data Fitted Networkscited by 47
  24. 2024What to align in multimodal contrastive learning?cited by 47
  25. 2025Griffin: Towards a Graph-Centric Relational Database Foundation Modelcited by 47
  26. 2023Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Eventscited by 46
  27. 2022Revisiting Pretraining Objectives for Tabular Deep Learningcited by 43
  28. 2021SurvTRACE: Transformers for Survival Analysis with Competing Eventscited by 41
  29. 2022T2G-Former: Organizing Tabular Features into Relation Graphs Promotes Heterogeneous Feature Interactioncited by 41
  30. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 41
  31. 2024SwitchTab: Switched Autoencoders Are Effective Tabular Learnerscited by 38
  32. 2024GenCAD: Image-Conditioned Computer-Aided Design Generation with Transformer-Based Contrastive Representation and Diffusion Priorscited by 38
  33. 2025LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizerscited by 38
  34. 2023Trompt: Towards a Better Deep Neural Network for Tabular Datacited by 36
  35. 2023HYTREL: Hypergraph-enhanced Tabular Data Representation Learningcited by 36
  36. 2025TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problemscited by 36
  37. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  38. 2023ExcelFormer: A neural network surpassing GBDTs on tabular datacited by 34

50 most cited of 740 All 740 (JSON)