# LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models

arXiv: https://arxiv.org/abs/2606.04485 · HTML: https://mlsanity.com/paper/2606.04485/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 16 of 16)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [OpenML Benchmarking Suites](https://mlsanity.com/paper/1708.03731/) (2017)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence](https://mlsanity.com/paper/2509.03505/) (2025)
- [Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models](https://mlsanity.com/paper/2510.21204/) (2025)

## Built on by (most cited first, 9 of 9)

- [LimiX-2: A Contextual Mechanism Network Towards General Structured-Data Intelligence](https://mlsanity.com/paper/2609.17488/) (2026)
- [A Fair Evaluation of Graph Foundation Models for Node Property Prediction](https://mlsanity.com/paper/2606.24509/) (2026)
- [GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models](https://mlsanity.com/paper/2608.18849/) (2026)
- [TabLoRA: Parameter-Efficient Low-Rank Ensemble Learning for Large-Scale Tabular Data](https://mlsanity.com/paper/2607.10077/) (2026)
- [SkillTFM: Gated Skill Evolution for Training-Free Adaptation of Tabular Foundation Models](https://mlsanity.com/paper/2608.06137/) (2026)
- [Fusion Anything: A Generalized Multimodal Foundation Model](https://mlsanity.com/paper/2609.22107/) (2026)
- [TAFFY: A Task-Adaptive Tabular Foundation Model with In-Context Diversity](https://mlsanity.com/paper/2610.07559/) (2026)
- [GeneICL: A Tabular Foundation Model for Bulk Transcriptomics](https://mlsanity.com/paper/2610.08694/) (2026)
- [TILE: Bridging LLMs and Table Foundation Models for Tabular Learning with Strings](https://mlsanity.com/paper/2610.17065/) (2026)

## Abstract (arXiv)

Tabular foundation models (TFMs) increasingly rival tree ensembles, but their performance is often compute-inefficient: with standard affine scalar tokenization, each feature injects value variation through an essentially one-dimensional channel, and feature IDs/positional signals cannot increase within-feature value degrees of freedom, yielding weak early-layer value sensitivity and redundant hidden states. We present a unified tokenize-and-route framework for strong TFMs: RaBEL expands each scalar into compact localized RBF features (optionally exponent-gated) to improve conditioning and shallow-layer effective rank, while a reordered bidirectional block S->N->F aligns computation with the readout by aggregating cross-sample context before feature mixing and using attention pooling. Together, these changes yield LimiX-2M, a 2M-parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used tabular benchmarks while reducing training and inference costs. These results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs. Model checkpoints and inference code are available at https://github.com/limix-ldm-ai/LimiX.