mlsanityData to 7 Oct 2026

Well-tuned Simple Nets Excel on Tabular Datasets

arXiv 2106.11189 ↗First posted 21 Jun 2021cs.LGcited by 151

Not on the map. Why?

Abstract (arXiv)

Tabular datasets are the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures. In this paper, we hypothesize that the key to boosting the performance of neural networks lies in rethinking the joint and simultaneous application of a large set of modern regularization techniques. As a result, we propose regularizing plain Multilayer Perceptron (MLP) networks by searching for the optimal combination/cocktail of 13 regularization techniques for each dataset using a joint optimization over the decision on which regularizers to apply and their subsidiary hyperparameters. We empirically assess the impact of these regularization cocktails for MLPs in a large-scale empirical study comprising 40 tabular datasets and demonstrate that (i) well-regularized plain MLPs significantly outperform recent state-of-the-art specialized neural network architectures, and (ii) they even outperform strong traditional ML methods, such as XGBoost.

Key facts. Cited by 151 papers in the map (data through 7 Oct 2026). Builds on 38 of its 68 references that are in the map. First posted on arXiv 21 Jun 2021.

Builds on

38 of 68 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
26 more
  1. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  2. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  3. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  5. 2017Deep Reinforcement Learning that Matterscited by 727
  6. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  7. 2019TabNet: Attentive Interpretable Tabular Learningcited by 640
  8. 2017Self-Normalizing Neural Networkscited by 591
  9. 2014OpenML: networked science in machine learningcited by 523
  10. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  11. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  12. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  13. 2020AutoGluon-Tabular: Robust and Accurate AutoML for Structured Datacited by 368
  14. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  15. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 333
  16. 2019Fast AutoAugmentcited by 243
  17. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 225
  18. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 200
  19. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 149
  20. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  21. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 103
  22. 2019An Open Source AutoML Benchmarkcited by 85
  23. 2020Auto-PyTorch Tabular: Multi-Fidelity MetaLearning for Efficient and Robust AutoDLcited by 67
  24. 2019OpenML-Python: an extensible Python API for OpenMLcited by 62
  25. 2020Neural Ensemble Search for Uncertainty Estimation and Dataset Shiftcited by 56
  26. 2018Bayesian Optimization in AlphaGocited by 55

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 27; 2023: 32; 2024: 39; 2025: 31; 2026: 21.

151 papers.

Sort:

By area:

  1. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  2. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  3. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  4. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 218
  5. 2022Why do tree-based models still outperform deep learning on tabular data?cited by 198
  6. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 185
  7. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  8. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  9. 2024TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensemblingcited by 105
  10. 2024Why Tabular Foundation Models Should Be a Research Prioritycited by 103
  11. 2024Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learningcited by 79
  12. 2023TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023cited by 73
38 more
  1. 2025Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Modelscited by 67
  2. 2023Benchmarking Distribution Shift in Tabular Data with TableShiftcited by 64
  3. 2022A Framework and Benchmark for Deep Batch Active Learning for Regressioncited by 47
  4. 2024TuneTables: Context Optimization for Scalable Prior-Data Fitted Networkscited by 47
  5. 2024TALENT: A Tabular Analytics and Learning Toolboxcited by 46
  6. 2022Revisiting Pretraining Objectives for Tabular Deep Learningcited by 43
  7. 2025A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilitiescited by 38
  8. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  9. 2023ExcelFormer: A neural network surpassing GBDTs on tabular datacited by 33
  10. 2023TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specializationcited by 32
  11. 2024Mixture of In-Context Prompters for Tabular PFNscited by 32
  12. 2024Revisiting Nearest Neighbor for Tabular Data: A Deep Tabular Baseline Two Decades Latercited by 28
  13. 2024Exploring Large Language Models for Feature Selection: A Data-centric Perspectivecited by 27
  14. 2023MediTab: Scaling Medical Tabular Data Predictors via Data Consolidation, Enrichment, and Refinementcited by 24
  15. 2024Privacy Preserving Prompt Engineering: A Surveycited by 24
  16. 2024HyperFast: Instant Classification for Tabular Datacited by 22
  17. 2024BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Modelcited by 21
  18. 2024A Survey on Deep Tabular Learningcited by 18
  19. 2022Subgroup Robustness Grows On Trees: An Empirical Baseline Investigationcited by 17
  20. 2022Learning Multimodal Data Augmentation in Feature Spacecited by 17
  21. 2024Text Serialization and Their Relationship with the Conventional Paradigms of Tabular Machine Learningcited by 17
  22. 2024A Survey on Self-Supervised Learning for Non-Sequential Tabular Datacited by 15
  23. 2023Beyond Individual Input for Deep Anomaly Detection on Tabular Datacited by 14
  24. 2024A Comprehensive Benchmark of Machine and Deep Learning Across Diverse Tabular Datasetscited by 13
  25. 2024T-JEPA: Augmentation-Free Self-Supervised Learning for Tabular Datacited by 13
  26. 2023Deep Clustering of Tabular Data by Weighted Gaussian Distribution Learningcited by 12
  27. 2023HyperTab: Hypernetwork Approach for Deep Learning on Small Tabular Datasetscited by 10
  28. 2023GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Datacited by 10
  29. 2022TractoSCR: A Novel Supervised Contrastive Regression Framework for Prediction of Neurocognitive Measures Using Multi-Site Harmonized Diffusion MRI Tractographycited by 9
  30. 2024Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarkingcited by 9
  31. 2024Transformers Meet Relational Databasescited by 9
  32. 2024Arithmetic Feature Interaction Is Necessary for Deep Tabular Learningcited by 8
  33. 2025Zero-shot Meta-learning for Tabular Prediction Tasks with Adversarially Pre-trained Transformercited by 8
  34. 2022Effectiveness of Deep Image Embedding Clustering Methods on Tabular Datacited by 7
  35. 2023Exploring Weight Balancing on Long-Tailed Recognition Problemcited by 7
  36. 2023A Comprehensive Survey on Generative Diffusion Models for Structured Datacited by 7
  37. 2025In-context Learning of Evolving Data Streams with Tabular Foundational Modelscited by 7
  38. 2025Prior-Fitted Networks Scale to Larger Datasets When Treated as Weak Learnerscited by 7

50 most cited of 151 All 151 (JSON)