# LoMETab: Beyond Rank-1 Ensembles for Tabular Deep Learning

arXiv: https://arxiv.org/abs/2605.14365 · HTML: https://mlsanity.com/paper/2605.14365/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 22 references that are in the map.
- First posted on arXiv 14 May 2026.

## Builds on (most cited first, 16 of 16)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning](https://mlsanity.com/paper/2002.06715/) (2020)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling](https://mlsanity.com/paper/2410.24210/) (2024)
- [Masksembles for Uncertainty Estimation](https://mlsanity.com/paper/2012.08334/) (2020)
- [TabR: Tabular Deep Learning Meets Nearest Neighbors in 2023](https://mlsanity.com/paper/2307.14338/) (2023)
- [TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks](https://mlsanity.com/paper/2406.19380/) (2024)
- [Diversity Matters When Learning From Ensembles](https://mlsanity.com/paper/2110.14149/) (2021)
- [FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation](https://mlsanity.com/paper/2206.00050/) (2022)
- [LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks](https://mlsanity.com/paper/2405.14438/) (2024)
- [Is BatchEnsemble a Single Model? On Calibration and Diversity of Efficient Ensembles](https://mlsanity.com/paper/2601.16936/) (2026)

## Abstract (arXiv)

Recent tabular learning benchmarks increasingly show a tight performance cluster rather than a clear hierarchy among leading methods, spanning gradient boosted decision trees, attention-based architectures, and implicit ensembles such as TabM. As benchmark gains plateau, a complementary goal is to understand and control the mechanisms that make simple neural tabular models competitive. We propose LoMETab, a rank-r generalization of multiplicative implicit ensembles. LoMETab lifts the rank-1 BatchEnsemble/TabM modulation to a rank-r identity-residual Hadamard family by parameterizing each member weight as Wk = W odot (1 + AkBk^top), where W is shared and (Ak, Bk) are member-specific low-rank factors. This exposes two practical diversity-control axes: the adapter rank r and the initialization scale σinit, and we prove that for r ≥ 2 this generalization strictly enlarges BatchEnsemble's hypothesis class. Empirically, we show that this added capacity manifests as measurable predictive diversity after training: on representative classification datasets, LoMETab sustains higher pairwise KL than an additive low-rank ablation, and (r, σinit) provides broad control over pairwise KL, varying by up to several orders of magnitude across configurations. The induced diversity is reflected in task-appropriate output-level measures: argmax disagreement for classification and ambiguity for regression, indicating that the control extends beyond pairwise KL to decision- and output-level member variation. Finally, experiments sweeping over adapter rank r and initialization scale σinit reveal that predictive performance is dataset-dependent over the (r, σinit) grid, supporting LoMETab as a controllable family of implicit ensembles rather than a fixed rank-1 construction.