# Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data

arXiv: https://arxiv.org/abs/2407.04491 · HTML: https://mlsanity.com/paper/2407.04491/

## Key facts

- Cited by 92 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2024.

## Built on by (most cited first, 30 of 92)

- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [TabDPT: Scaling Tabular Foundation Models on Real Data](https://mlsanity.com/paper/2410.18164/) (2024)
- [TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling](https://mlsanity.com/paper/2410.24210/) (2024)
- [Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models](https://mlsanity.com/paper/2510.21204/) (2025)
- [MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement](https://mlsanity.com/paper/2506.15692/) (2025)
- [A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities](https://mlsanity.com/paper/2502.17361/) (2025)
- [TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems](https://mlsanity.com/paper/2502.02527/) (2025)
- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [ConTextTab: A Semantics-Aware Tabular In-Context Learner](https://mlsanity.com/paper/2506.10707/) (2025)
- [Table Foundation Models: on knowledge pre-training for tabular learning](https://mlsanity.com/paper/2505.14415/) (2025)
- [TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields](https://mlsanity.com/paper/2505.18125/) (2025)
- [Rethinking Early Stopping: Refine, Then Calibrate](https://mlsanity.com/paper/2501.19195/) (2025)
- [xRFM: Accurate, scalable, and interpretable feature learning models for tabular data](https://mlsanity.com/paper/2508.10053/) (2025)
- [TabPFN-Wide: Continued Pre-Training for Extreme Feature Counts](https://mlsanity.com/paper/2510.06162/) (2025)
- [Relational In-Context Learning via Synthetic Pre-training with Structural Prior](https://mlsanity.com/paper/2603.03805/) (2026)
- [On Finetuning Tabular Foundation Models](https://mlsanity.com/paper/2506.08982/) (2025)
- [EXAONE Tabular 1.0 : Technical Report](https://mlsanity.com/paper/2608.25774/) (2026)
- [Conditional Coverage Diagnostics for Conformal Prediction](https://mlsanity.com/paper/2512.11779/) (2025)
- [TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention](https://mlsanity.com/paper/2606.07345/) (2026)
- [LimiX-2: A Contextual Mechanism Network Towards General Structured-Data Intelligence](https://mlsanity.com/paper/2609.17488/) (2026)
- [Position: Foundation Models for Tabular Data within Systemic Contexts Need Grounding](https://mlsanity.com/paper/2505.19825/) (2025)
- [TabGemma: Text-Based Tabular ICL via LLM using Continued Pretraining and Retrieval](https://mlsanity.com/paper/2511.03570/) (2025)
- [ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules](https://mlsanity.com/paper/2603.29928/) (2026)
- [The Credibility Transformer](https://mlsanity.com/paper/2409.16653/) (2024)
- [Understanding the Limits of Deep Tabular Methods with Temporal Shift](https://mlsanity.com/paper/2502.20260/) (2025)
- [Benchmarking Optimizers for MLPs in Tabular Deep Learning](https://mlsanity.com/paper/2604.15297/) (2026)
- [Put CASH on Bandits: A Max K-Armed Problem for Automated Machine Learning](https://mlsanity.com/paper/2505.05226/) (2025)
- [Realistic Evaluation of TabPFN v2 in Open Environments](https://mlsanity.com/paper/2505.16226/) (2025)
- [Modern Neural Networks for Small Tabular Datasets: The New Default for Field-Scale Digital Soil Mapping?](https://mlsanity.com/paper/2508.09888/) (2025)
- [Mitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustment](https://mlsanity.com/paper/2605.04363/) (2026)

## Abstract (arXiv)

For classification and regression on tabular data, the dominance of gradient-boosted decision trees (GBDTs) has recently been challenged by often much slower deep learning methods with extensive hyperparameter tuning. We address this discrepancy by introducing (a) RealMLP, an improved multilayer perceptron (MLP), and (b) strong meta-tuned default parameters for GBDTs and RealMLP. We tune RealMLP and the default parameters on a meta-train benchmark with 118 datasets and compare them to hyperparameter-optimized versions on a disjoint meta-test benchmark with 90 datasets, as well as the GBDT-friendly benchmark by Grinsztajn et al. (2022). Our benchmark results on medium-to-large tabular datasets (1K--500K samples) show that RealMLP offers a favorable time-accuracy tradeoff compared to other neural baselines and is competitive with GBDTs in terms of benchmark scores. Moreover, a combination of RealMLP and GBDTs with improved default parameters can achieve excellent results without hyperparameter tuning. Finally, we demonstrate that some of RealMLP's improvements can also considerably improve the performance of TabR with default parameters.