# TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks

arXiv: https://arxiv.org/abs/2406.19380 · HTML: https://mlsanity.com/paper/2406.19380/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 50 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 30 of 32)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations](https://mlsanity.com/paper/2204.02937/) (2022)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption](https://mlsanity.com/paper/2106.15147/) (2021)
- [Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning](https://mlsanity.com/paper/2106.02584/) (2021)
- [Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks](https://mlsanity.com/paper/2107.07455/) (2021)
- [Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Time](https://mlsanity.com/paper/2211.14238/) (2022)
- [TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023](https://mlsanity.com/paper/2307.14338/) (2023)
- [Benchmarking Distribution Shift in Tabular Data with TableShift](https://mlsanity.com/paper/2312.07577/) (2023)
- [TuneTables: Context Optimization for Scalable Prior-Data Fitted Networks](https://mlsanity.com/paper/2402.11137/) (2024)
- [Revisiting Pretraining Objectives for Tabular Deep Learning](https://mlsanity.com/paper/2207.03208/) (2022)
- [Trompt: Towards a Better Deep Neural Network for Tabular Data](https://mlsanity.com/paper/2305.18446/) (2023)
- [On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models](https://mlsanity.com/paper/2209.05310/) (2022)
- [ExcelFormer: A neural network surpassing GBDTs on tabular data](https://mlsanity.com/paper/2301.02819/) (2023)
- [TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specialization](https://mlsanity.com/paper/2303.05506/) (2023)
- [A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication](https://mlsanity.com/paper/2111.04424/) (2021)
- [Binning as a Pretext Task: Improving Self-Supervised Learning in Tabular Domains](https://mlsanity.com/paper/2405.07414/) (2024)
- [DeDrift: Robust Similarity Search under Content Drift](https://mlsanity.com/paper/2308.02752/) (2023)
- [Wild-Tab: A Benchmark For Out-Of-Distribution Generalization In Tabular Regression](https://mlsanity.com/paper/2312.01792/) (2023)

## Built on by (most cited first, 30 of 33)

- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling](https://mlsanity.com/paper/2410.24210/) (2024)
- [Beyond IID: How General Are Tabular Foundation Models, Really?](https://mlsanity.com/paper/2606.30410/) (2026)
- [On Finetuning Tabular Foundation Models](https://mlsanity.com/paper/2506.08982/) (2025)
- [Benchmarking Optimizers for MLPs in Tabular Deep Learning](https://mlsanity.com/paper/2604.15297/) (2026)
- [Modern Neural Networks for Small Tabular Datasets: The New Default for Field-Scale Digital Soil Mapping?](https://mlsanity.com/paper/2508.09888/) (2025)
- [LakeMLB: Data Lake Machine Learning Benchmark](https://mlsanity.com/paper/2602.10441/) (2026)
- [Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules](https://mlsanity.com/paper/2603.08206/) (2026)
- [Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression](https://mlsanity.com/paper/2603.26611/) (2026)
- [MultiTab: A Comprehensive Benchmark Suite for Multi-Dimensional Evaluation in Tabular Domains](https://mlsanity.com/paper/2505.14312/) (2025)
- [Unveiling the Role of Data Uncertainty in Tabular Deep Learning](https://mlsanity.com/paper/2509.04430/) (2025)
- [TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks](https://mlsanity.com/paper/2606.02384/) (2026)
- [iLTM: Integrated Large Tabular Model](https://mlsanity.com/paper/2511.15941/) (2025)
- [OmniTabBench: Mapping the Empirical Frontiers of GBDTs, Neural Networks, and Foundation Models for Tabular Data at Scale](https://mlsanity.com/paper/2604.06814/) (2026)
- [DynaTab: Dynamic Feature Ordering as Neural Rewiring for High-Dimensional Tabular Data](https://mlsanity.com/paper/2605.03430/) (2026)
- [BoostLLM: Boosting-inspired LLM Fine-tuning for Few-shot Tabular Classification](https://mlsanity.com/paper/2605.06117/) (2026)
- [Exploring Differences Between Tabular Enterprise Data and Public Benchmarks](https://mlsanity.com/paper/2606.30452/) (2026)
- [Inverse Entropic Optimal Transport Solves Semi-supervised Learning via Data Likelihood Maximization](https://mlsanity.com/paper/2410.02628/) (2024)
- [ICE-ID: A Novel Historical Census Dataset for Longitudinal Identity Resolution](https://mlsanity.com/paper/2506.13792/) (2025)
- [Strategic inputs: feature selection from game-theoretic perspective](https://mlsanity.com/paper/2510.24982/) (2025)
- [Aeolus: A Multi-structural Flight Delay Dataset](https://mlsanity.com/paper/2510.26616/) (2025)
- [Automated Model Design using Gated Neuron Selection in Telecom](https://mlsanity.com/paper/2602.10854/) (2026)
- [Vectorized Dynamic Histograms for Sparse Oblique Forests](https://mlsanity.com/paper/2603.00326/) (2026)
- [LassoFlexNet: Flexible Neural Architecture for Tabular Data](https://mlsanity.com/paper/2603.20631/) (2026)
- [Analog Optical Inference on Million-Record Mortgage Data](https://mlsanity.com/paper/2604.13251/) (2026)
- [Data Language Models: A New Foundation Model Class for Tabular Data](https://mlsanity.com/paper/2605.06290/) (2026)
- [Hedging Memory Horizons for Non-Stationary Prediction via Online Aggregation](https://mlsanity.com/paper/2605.06541/) (2026)
- [LoMETab: Beyond Rank-1 Ensembles for Tabular Deep Learning](https://mlsanity.com/paper/2605.14365/) (2026)
- [InFlux++: Real and Synthetic Data for Estimating Dynamic Camera Intrinsics](https://mlsanity.com/paper/2607.05389/) (2026)
- [Agentic Search Spaces for Tabular Machine Learning](https://mlsanity.com/paper/2609.16309/) (2026)

## Abstract (arXiv)

Advances in machine learning research drive progress in real-world applications. To ensure this progress, it is important to understand the potential pitfalls on the way from a novel method's success on academic benchmarks to its practical deployment. In this work, we analyze existing tabular benchmarks and find two common characteristics of tabular data in typical industrial applications that are underrepresented in the datasets usually used for evaluation in the literature. First, in real-world deployment scenarios, distribution of data often changes over time. To account for this distribution drift, time-based train/test splits should be used in evaluation. However, popular tabular datasets often lack timestamp metadata to enable such evaluation. Second, a considerable portion of datasets in production settings stem from extensive data acquisition and feature engineering pipelines. This can have an impact on the absolute and relative number of predictive, uninformative, and correlated features compared to academic datasets. In this work, we aim to understand how recent research advances in tabular deep learning transfer to these underrepresented conditions. To this end, we introduce TabReD -- a collection of eight industry-grade tabular datasets. We reassess a large number of tabular ML models and techniques on TabReD. We demonstrate that evaluation on time-based data splits leads to different methods ranking, compared to evaluation on random splits, which are common in current benchmarks. Furthermore, simple MLP-like architectures and GBDT show the best results on the TabReD datasets, while other methods are less effective in the new setting.