# A Closer Look at Deep Learning Methods on Tabular Datasets

arXiv: https://arxiv.org/abs/2407.00956 · HTML: https://mlsanity.com/paper/2407.00956/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jul 2024.

## Built on by (most cited first, 30 of 41)

- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [ConTextTab: A Semantics-Aware Tabular In-Context Learner](https://mlsanity.com/paper/2506.10707/) (2025)
- [Beyond IID: How General Are Tabular Foundation Models, Really?](https://mlsanity.com/paper/2606.30410/) (2026)
- [Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning](https://mlsanity.com/paper/2511.02818/) (2025)
- [Rethinking Early Stopping: Refine, Then Calibrate](https://mlsanity.com/paper/2501.19195/) (2025)
- [A Comprehensive Benchmark of Machine and Deep Learning Across Diverse Tabular Datasets](https://mlsanity.com/paper/2408.14817/) (2024)
- [TabTune: A Unified Library for Inference and Fine-Tuning Tabular Foundation Models](https://mlsanity.com/paper/2511.02802/) (2025)
- [Orion-Bix: Bi-Axial Attention for Tabular In-Context Learning](https://mlsanity.com/paper/2512.00181/) (2025)
- [Exploring Fine-Tuning for Tabular Foundation Models](https://mlsanity.com/paper/2601.09654/) (2026)
- [TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention](https://mlsanity.com/paper/2606.07345/) (2026)
- [Position: Foundation Models for Tabular Data within Systemic Contexts Need Grounding](https://mlsanity.com/paper/2505.19825/) (2025)
- [Understanding the Limits of Deep Tabular Methods with Temporal Shift](https://mlsanity.com/paper/2502.20260/) (2025)
- [The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Tasks](https://mlsanity.com/paper/2506.08274/) (2025)
- [Modern Neural Networks for Small Tabular Datasets: The New Default for Field-Scale Digital Soil Mapping?](https://mlsanity.com/paper/2508.09888/) (2025)
- [STRABLE: Benchmarking Tabular Machine Learning with Strings](https://mlsanity.com/paper/2605.12292/) (2026)
- [Mitra-v2 Technical Report](https://mlsanity.com/paper/2609.04540/) (2026)
- [On the retraining frequency of global models in retail demand forecasting](https://mlsanity.com/paper/2505.00356/) (2025)
- [MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretraining](https://mlsanity.com/paper/2509.06806/) (2025)
- [No Data? No Problem: Robust Vision-Tabular Learning with Missing Values](https://mlsanity.com/paper/2512.19602/) (2025)
- [Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression](https://mlsanity.com/paper/2603.26611/) (2026)
- [Relatron: Automating Relational Machine Learning over Relational Databases](https://mlsanity.com/paper/2602.22552/) (2026)
- [Selecting Feature Interactions for Generalized Additive Models by Distilling Foundation Models](https://mlsanity.com/paper/2604.13332/) (2026)
- [TopBench: A Benchmark for Implicit Predictive Reasoning in Tabular Question Answering](https://mlsanity.com/paper/2604.28076/) (2026)
- [MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image](https://mlsanity.com/paper/2605.10616/) (2026)
- [Feature-aware Modulation for Learning from Temporal Tabular Data](https://mlsanity.com/paper/2512.03678/) (2025)
- [Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning](https://mlsanity.com/paper/2512.06533/) (2025)
- [Predicting Anemia Among Under-Five Children in Nepal Using Machine Learning and Deep Learning](https://mlsanity.com/paper/2602.01005/) (2026)
- [BoostLLM: Boosting-inspired LLM Fine-tuning for Few-shot Tabular Classification](https://mlsanity.com/paper/2605.06117/) (2026)
- [Explaining Tabular Foundation Model Differences Through Meta-Features](https://mlsanity.com/paper/2605.28418/) (2026)
- [SOMTab: Set-Order Mamba for Efficient Tabular In-Context Learning](https://mlsanity.com/paper/2608.27882/) (2026)

## Abstract (arXiv)

Tabular data is prevalent across diverse domains in machine learning. With the rapid progress of deep tabular prediction methods, especially pretrained (foundation) models, there is a growing need to evaluate these methods systematically and to understand their behavior. We present an extensive study on TALENT, a collection of 300+ datasets spanning broad ranges of size, feature composition (numerical/categorical mixes), domains, and output types (binary, multi--class, regression). Our evaluation shows that ensembling benefits both tree-based and neural approaches. Traditional gradient-boosted trees remain very strong baselines, yet recent pretrained tabular models now match or surpass them on many tasks, narrowing--but not eliminating--the historical advantage of tree ensembles. Despite architectural diversity, top performance concentrates within a small subset of models, providing practical guidance for method selection. To explain these outcomes, we quantify dataset heterogeneity by learning from meta-features and early training dynamics to predict later validation behavior. This dynamics-aware analysis indicates that heterogeneity--such as the interplay of categorical and numerical attributes--largely determines which family of methods is favored. Finally, we introduce a two-level design beyond the 300 common-size datasets: a compact TALENT-tiny core (45 datasets) for rapid, reproducible evaluation, and a TALENT-extension suite targeting high-dimensional, many-class, and very large-scale settings for stress testing. In summary, these results offer actionable insights into the strengths, limitations, and future directions for improving deep tabular learning.