Understanding the Surprising Generalization Properties of Tabular Foundation Models
Not on the map. Why?
Abstract (arXiv)
Tabular Foundation Models (TFMs) increasingly rely on in-context learning, where a model receives labelled examples at inference time and predicts labels for new inputs without updating its weights. Existing TFMs are typically trained on either massive synthetic corpora or very large collections of real datasets. In contrast, we show that surprisingly strong transfer can emerge from self-supervised pre-training on just a single real table. In this setting, we also find that tables tend to be either broadly useful or broadly poor regardless of downstream prediction task, and that the strongest predictor of usefulness is the number of features rather than the number of instances. This leads to a task-centric interpretation of tabular pre-training: the number and the quality of tasks are essential for the pre-training of TFMs. We show that the same task-centric perspective can help corpus design at scale: fine-grained column-level pre-processing consistently improves downstream performance, while no improvements are observed when we filter or deduplicate at the dataset level. Finally, we offer a new perspective for how TFMs generalize: we believe that tabular in-context generalization is largely retrieval-based, and good models are those that learn to identify relevant examples in the provided context and aggregate them well. The mechanics of TFMs have been relatively understudied; our task-centric, retrieval-based perspective offers a new framework to guide future model and corpus design.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 18 Aug 2026.
Builds on
25 of 35 references.
References · 25
- 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
- 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
- 2021Deduplicating Training Data Makes Language Models Bettercited by 714
- 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
- 2021Transformers Can Do Bayesian Inferencecited by 360
- 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
- 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 289
- 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
- 2017OpenML Benchmarking Suitescited by 178
- 2025TabArena: A Living Benchmark for Machine Learning on Tabular Datacited by 176
- 2026TabICLv2: A better, faster, scalable, and open tabular foundation modelcited by 144
13 more
- 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 118
- 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 113
- 2025LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligencecited by 79
- 2025Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Modelscited by 77
- 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 76
- 2024Retrieval & Fine-Tuning for In-Context Tabular Modelscited by 59
- 2025Do-PFN: In-Context Learning for Causal Effect Estimationcited by 51
- 2025CausalPFN: Amortized Causal Effect Estimation via In-Context Learningcited by 49
- 2025Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Datacited by 41
- 2025ConTextTab: A Semantics-Aware Tabular In-Context Learnercited by 34
- 2024Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Datacited by 25
- 2026TabDPT-Turbo: Efficient In-Context Learning for Tabular Predictioncited by 6
- 2026Do Tabular Foundation Models Agree with Themselves?cited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.