mlsanityData to 14 Oct 2026

When Do Neural Nets Outperform Boosted Trees on Tabular Data?

arXiv 2305.02997 ↗First posted 4 May 2023cs.LG cs.AI stat.MLcited by 223

Not on the map. Why?

Abstract (arXiv)

Tabular data is one of the most commonly used types of data in machine learning. Despite recent advances in neural nets (NNs) for tabular data, there is still an active discussion on whether or not NNs generally outperform gradient-boosted decision trees (GBDTs) on tabular data, with several recent works arguing either that GBDTs consistently outperform NNs on tabular data, or vice versa. In this work, we take a step back and question the importance of this debate. To this end, we conduct the largest tabular data analysis to date, comparing 19 algorithms across 176 datasets, and we find that the 'NN vs. GBDT' debate is overemphasized: for a surprisingly high number of datasets, either the performance difference between GBDTs and NNs is negligible, or light hyperparameter tuning on a GBDT is more important than choosing between NNs and GBDTs. A remarkable exception is the recently-proposed prior-data fitted network, TabPFN: although it is effectively limited to training sets of size 3000, we find that it outperforms all other algorithms on average, even when randomly sampling 3000 training datapoints. Next, we analyze dozens of metafeatures to determine what properties of a dataset make NNs or GBDTs better-suited to perform well. For example, we find that GBDTs are much better than NNs at handling skewed or heavy-tailed feature distributions and other forms of dataset irregularities. Our insights act as a guide for practitioners to determine which techniques may work best on their dataset. Finally, with the goal of accelerating tabular data research, we release the TabZilla Benchmark Suite: a collection of the 36 'hardest' of the datasets we study. Our benchmark suite, codebase, and all raw results are available at https://github.com/naszilla/tabzilla.

Key facts. Cited by 223 papers in the map (data through 14 Oct 2026). Builds on 33 of its 74 references that are in the map. First posted on arXiv 4 May 2023.

Builds on

33 of 74 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  7. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  8. 2018Datasheets for Datasetscited by 1,386
  9. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  10. 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 676
  11. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  12. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
21 more
  1. 2021Tabular Data: Deep Learning is Not All You Needcited by 535
  2. 2014OpenML: networked science in machine learningcited by 526
  3. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
  4. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  5. 2021Transformers Can Do Bayesian Inferencecited by 360
  6. 2020Neural Additive Models: Interpretable Machine Learning with Neural Netscited by 319
  7. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 319
  8. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 277
  9. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 203
  10. 2017OpenML Benchmarking Suitescited by 178
  11. 2020Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learningcited by 174
  12. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  13. 2018Troubling Trends in Machine Learning Scholarshipcited by 108
  14. 2019An Open Source AutoML Benchmarkcited by 85
  15. 2020Trust Issues: Uncertainty Estimation Does Not Enable Reliable OOD Detection On Medical Tabular Datacited by 52
  16. 2022Transfer Learning with Deep Tabular Modelscited by 52
  17. 2021DANets: Deep Abstract Networks for Tabular Data Classification and Regressioncited by 50
  18. 2022Revisiting Pretraining Objectives for Tabular Deep Learningcited by 43
  19. 2020Sequential Deep Learning for Credit Risk Monitoring with Tabular Financial Datacited by 39
  20. 2023TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specializationcited by 32
  21. 2022Hopular: Modern Hopfield Networks for Tabular Datacited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 59; 2025: 77; 2026: 73.

223 papers.

Sort:

By area:

  1. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 113
  2. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 106
  3. 2023ForecastPFN: Synthetically-Trained Zero-Shot Forecastingcited by 104
  4. 2025Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Modelscited by 77
  5. 2024CARTE: Pretraining and Transfer for Tabular Learningcited by 76
  6. 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 76
  7. 2024Retrieval & Fine-Tuning for In-Context Tabular Modelscited by 59
  8. 2024Amazing Things Come From Having Many Good Modelscited by 55
  9. 2024TALENT: A Tabular Analytics and Learning Toolboxcited by 52
  10. 2025Do-PFN: In-Context Learning for Causal Effect Estimationcited by 51
  11. 2024Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoningcited by 49
  12. 2025CausalPFN: Amortized Causal Effect Estimation via In-Context Learningcited by 49
38 more
  1. 2024TuneTables: Context Optimization for Scalable Prior-Data Fitted Networkscited by 47
  2. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 41
  3. 2025TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problemscited by 36
  4. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  5. 2023TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applicationscited by 33
  6. 2024Mixture of In-Context Prompters for Tabular PFNscited by 33
  7. 2024TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarkscited by 33
  8. 2024Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Latercited by 30
  9. 2024HyperFast: Instant Classification for Tabular Datacited by 22
  10. 2024BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Modelcited by 21
  11. 2024Mambular: A Sequential Model for Tabular Deep Learningcited by 21
  12. 2025Position: The Future of Bayesian Prediction Is Prior-Fittedcited by 20
  13. 2023MotherNet: Fast Training and Inference via Hyper-Network Transformerscited by 19
  14. 2024In-Context Data Distillation with TabPFNcited by 19
  15. 2024Position: Why We Must Rethink Empirical Research in Machine Learningcited by 19
  16. 2025Table Foundation Models: on knowledge pre-training for tabular learningcited by 19
  17. 2025Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learningcited by 19
  18. 2023ProtoGate: Prototype-based Neural Networks with Global-to-local Feature Selection for Tabular Biomedical Datacited by 18
  19. 2023Scaling TabPFN: Sketching and Feature Selection for Tabular Prior-Data Fitted Networkscited by 18
  20. 2023Simplifying Neural Network Training Under Class Imbalancecited by 18
  21. 2025TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fieldscited by 18
  22. 2025Rethinking Early Stopping: Refine, Then Calibratecited by 17
  23. 2025Operator Learning: A Statistical Perspectivecited by 17
  24. 2024TabEBM: A Tabular Data Augmentation Method with Distinct Class-Specific Energy-Based Modelscited by 16
  25. 2024A Survey on Self-Supervised Learning for Non-Sequential Tabular Datacited by 15
  26. 2024A Data-Centric Perspective on Evaluating Machine Learning Models for Tabular Datacited by 15
  27. 2024A Comprehensive Benchmark of Machine and Deep Learning Across Diverse Tabular Datasetscited by 15
  28. 2025TreeLUT: An Efficient Alternative to Deep Neural Networks for Inference Acceleration Using Gradient Boosted Decision Treescited by 15
  29. 2025TabFlex: Scaling Tabular Learning to Millions with Linear Attentioncited by 15
  30. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  31. 2024Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domainscited by 12
  32. 2023GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Datacited by 11
  33. 2024Interpretable Machine Learning for TabPFNcited by 11
  34. 2025TabTune: A Unified Library for Inference and Fine-Tuning Tabular Foundation Modelscited by 11
  35. 2024HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Modelscited by 10
  36. 2024Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyondcited by 9
  37. 2026Exploring Fine-Tuning for Tabular Foundation Modelscited by 9
  38. 2026From Zero to Hero: Advancing Zero-Shot Foundation Models for Tabular Outlier Detectioncited by 9

50 most cited of 223 All 223 (JSON)