# TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling

arXiv: https://arxiv.org/abs/2410.24210 · HTML: https://mlsanity.com/paper/2410.24210/

## Key facts

- Cited by 113 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 50 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://mlsanity.com/paper/1810.11921/) (2018)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning](https://mlsanity.com/paper/2012.09816/) (2020)
- [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](https://mlsanity.com/paper/1909.06312/) (2019)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [Training independent subnetworks for robust prediction](https://mlsanity.com/paper/2010.06610/) (2020)
- [SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption](https://mlsanity.com/paper/2106.15147/) (2021)
- [Well-tuned Simple Nets Excel on Tabular Datasets](https://mlsanity.com/paper/2106.11189/) (2021)
- [Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks](https://mlsanity.com/paper/1511.06314/) (2015)
- [Introducing LETOR 4.0 Datasets](https://mlsanity.com/paper/1306.2597/) (2013)
- [Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning](https://mlsanity.com/paper/2106.02584/) (2021)
- [Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data](https://mlsanity.com/paper/2407.04491/) (2024)
- [CARTE: Pretraining and Transfer for Tabular Learning](https://mlsanity.com/paper/2402.16785/) (2024)
- [TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023](https://mlsanity.com/paper/2307.14338/) (2023)
- [Depth Uncertainty in Neural Networks](https://mlsanity.com/paper/2006.08437/) (2020)
- [Packed-Ensembles for Efficient Uncertainty Estimation](https://mlsanity.com/paper/2210.09184/) (2022)

## Built on by (most cited first, 30 of 113)

- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models](https://mlsanity.com/paper/2510.21204/) (2025)
- [A Closer Look at TabPFN v2: Understanding Its Strengths and Extending Its Capabilities](https://mlsanity.com/paper/2502.17361/) (2025)
- [TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problems](https://mlsanity.com/paper/2502.02527/) (2025)
- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [Beyond IID: How General Are Tabular Foundation Models, Really?](https://mlsanity.com/paper/2606.30410/) (2026)
- [TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields](https://mlsanity.com/paper/2505.18125/) (2025)
- [xRFM: Accurate, scalable, and interpretable feature learning models for tabular data](https://mlsanity.com/paper/2508.10053/) (2025)
- [Relational In-Context Learning via Synthetic Pre-training with Structural Prior](https://mlsanity.com/paper/2603.03805/) (2026)
- [On Finetuning Tabular Foundation Models](https://mlsanity.com/paper/2506.08982/) (2025)
- [Chunked TabPFN: Exact Training-Free In-Context Learning for Long-Context Tabular Data](https://mlsanity.com/paper/2509.00326/) (2025)
- [TabSwift: An Efficient Tabular Foundation Model with Row-Wise Attention](https://mlsanity.com/paper/2606.07345/) (2026)
- [LimiX-2: A Contextual Mechanism Network Towards General Structured-Data Intelligence](https://mlsanity.com/paper/2609.17488/) (2026)
- [TabGemma: Text-Based Tabular ICL via LLM using Continued Pretraining and Retrieval](https://mlsanity.com/paper/2511.03570/) (2025)
- [End-to-End Compression for Tabular Foundation Models](https://mlsanity.com/paper/2602.05649/) (2026)
- [ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules](https://mlsanity.com/paper/2603.29928/) (2026)
- [TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction](https://mlsanity.com/paper/2608.01400/) (2026)
- [TabFM: A Zero-Shot Foundation Model for Tabular Data](https://mlsanity.com/paper/2609.37959/) (2026)
- [Understanding the Limits of Deep Tabular Methods with Temporal Shift](https://mlsanity.com/paper/2502.20260/) (2025)
- [Benchmarking Optimizers for MLPs in Tabular Deep Learning](https://mlsanity.com/paper/2604.15297/) (2026)
- [Modern Neural Networks for Small Tabular Datasets: The New Default for Field-Scale Digital Soil Mapping?](https://mlsanity.com/paper/2508.09888/) (2025)
- [Understanding Structured Financial Data with LLMs: A Case Study on Fraud Detection](https://mlsanity.com/paper/2512.13040/) (2025)
- [Retrieval-aligned Tabular Foundation Models Enable Robust Clinical Risk Prediction in Electronic Health Records Under Real-world Constraints](https://mlsanity.com/paper/2604.01841/) (2026)
- [Mitigating Label Shift in Tabular In-Context Learning via Test-Time Posterior Adjustment](https://mlsanity.com/paper/2605.04363/) (2026)
- [Mitra-v2 Technical Report](https://mlsanity.com/paper/2609.04540/) (2026)
- [Make Still Further Progress: Chain of Thoughts for Tabular Data Leaderboard](https://mlsanity.com/paper/2505.13421/) (2025)
- [Beyond ReLU: How Activations Affect Neural Kernels and Random Wide Networks](https://mlsanity.com/paper/2506.22429/) (2025)
- [TabDistill: Distilling Transformers into Neural Nets for Few-Shot Tabular Classification](https://mlsanity.com/paper/2511.05704/) (2025)
- [Causal Data Augmentation for Robust Fine-Tuning of Tabular Foundation Models](https://mlsanity.com/paper/2601.04110/) (2026)

## Abstract (arXiv)

Deep learning architectures for supervised learning on tabular data range from simple multilayer perceptrons (MLP) to sophisticated Transformers and retrieval-augmented methods. This study highlights a major, yet so far overlooked opportunity for designing substantially better MLP-based tabular architectures. Namely, our new model TabM relies on efficient ensembling, where one TabM efficiently imitates an ensemble of MLPs and produces multiple predictions per object. Compared to a traditional deep ensemble, in TabM, the underlying implicit MLPs are trained simultaneously, and (by default) share most of their parameters, which results in significantly better performance and efficiency. Using TabM as a new baseline, we perform a large-scale evaluation of tabular DL architectures on public benchmarks in terms of both task performance and efficiency, which renders the landscape of tabular DL in a new light. Generally, we show that MLPs, including TabM, form a line of stronger and more practical models compared to attention- and retrieval-based architectures. In particular, we find that TabM demonstrates the best performance among tabular DL models. Then, we conduct an empirical analysis on the ensemble-like nature of TabM. We observe that the multiple predictions of TabM are weak individually, but powerful collectively. Overall, our work brings an impactful technique to tabular DL and advances the performance-efficiency trade-off with TabM -- a simple and powerful baseline for researchers and practitioners.