# Table Foundation Models: on knowledge pre-training for tabular learning

arXiv: https://arxiv.org/abs/2505.14415 · HTML: https://mlsanity.com/paper/2505.14415/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 64 references that are in the map.
- First posted on arXiv 20 May 2025.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [On the Bottleneck of Graph Neural Networks and its Practical Implications](https://mlsanity.com/paper/2006.05205/) (2020)
- [Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View](https://mlsanity.com/paper/1909.03211/) (2019)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [Matryoshka Representation Learning](https://mlsanity.com/paper/2205.13147/) (2022)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [Advances in Pre-Training Distributed Word Representations](https://mlsanity.com/paper/1712.09405/) (2017)
- [Challenges in Deploying Machine Learning: a Survey of Case Studies](https://mlsanity.com/paper/2011.09926/) (2020)
- [LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks](https://mlsanity.com/paper/2206.06565/) (2022)
- [TableLlama: Towards Open Large Generalist Models for Tables](https://mlsanity.com/paper/2311.09206/) (2023)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data](https://mlsanity.com/paper/2407.04491/) (2024)
- [XTab: Cross-table Pretraining for Tabular Transformers](https://mlsanity.com/paper/2305.06090/) (2023)
- [CARTE: Pretraining and Transfer for Tabular Learning](https://mlsanity.com/paper/2402.16785/) (2024)
- [Large Scale Transfer Learning for Tabular Data via Language Modeling](https://mlsanity.com/paper/2406.12031/) (2024)

## Built on by (most cited first, 19 of 19)

- [Beyond IID: How General Are Tabular Foundation Models, Really?](https://mlsanity.com/paper/2606.30410/) (2026)
- [Survival In-Context: Amortized Bayesian Survival Analysis via Prior-Fitted Networks](https://mlsanity.com/paper/2603.29475/) (2026)
- [Towards Universal Tabular Embeddings: A Benchmark Across Data Tasks](https://mlsanity.com/paper/2604.21696/) (2026)
- [TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Models](https://mlsanity.com/paper/2506.18421/) (2025)
- [No Data? No Problem: Robust Vision-Tabular Learning with Missing Values](https://mlsanity.com/paper/2512.19602/) (2025)
- [RamanBench: A Large-Scale Benchmark for Machine Learning on Raman Spectroscopy](https://mlsanity.com/paper/2605.02003/) (2026)
- [Grables: When Graph Structure Helps Tabular Prediction](https://mlsanity.com/paper/2602.03945/) (2026)
- [MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image](https://mlsanity.com/paper/2605.10616/) (2026)
- [Universal Embeddings of Tabular Data](https://mlsanity.com/paper/2507.05904/) (2025)
- [JT-DA: Enhancing Data Analysis with Tool-Integrated Table Reasoning Large Language Models](https://mlsanity.com/paper/2512.06859/) (2025)
- [TabSurv: Adapting Modern Tabular Neural Networks to Survival Analysis](https://mlsanity.com/paper/2605.03944/) (2026)
- [Fair Classification with Efficient and Post-hoc Controllable Fairness-Accuracy Trade-off](https://mlsanity.com/paper/2606.28097/) (2026)
- [The Importance of Encoder Choice:A Tabular-Image Study](https://mlsanity.com/paper/2607.07756/) (2026)
- [TabuLM: Morphology-Aware Tabular Pre-training for Low-Resource Languages](https://mlsanity.com/paper/2608.26923/) (2026)
- [Generalized Context in Cross Attention for Transfer Learning of Disjoint Tabular Data](https://mlsanity.com/paper/2608.28209/) (2026)
- [Whole-Body MRI Classification via Prompt-Based Clinical Conditioning](https://mlsanity.com/paper/2608.30824/) (2026)
- [Beyond Correctness: Evaluating Semantic Knowledge in Cross-Table Transfer](https://mlsanity.com/paper/2609.34098/) (2026)
- [EchoTab: Learning Measurement-Grounded Representations for Echocardiographic Diagnosis through Tabular Reconstruction](https://mlsanity.com/paper/2610.13599/) (2026)
- [TILE: Bridging LLMs and Table Foundation Models for Tabular Learning with Strings](https://mlsanity.com/paper/2610.17065/) (2026)

## Abstract (arXiv)

Table foundation models bring high hopes to data science: pre-trained on tabular data to embark knowledge or priors, they should facilitate downstream tasks on tables. One specific challenge is that of data semantics: numerical entries take their meaning from context, e.g., column name. Pre-trained neural networks that jointly model column names and table entries have recently boosted prediction accuracy. While these models outline the promises of world knowledge to interpret table values, they lack the convenience of popular foundation models in text or vision. Indeed, they must be fine-tuned to bring benefits, come with sizeable computation costs, and cannot easily be reused or combined with other architectures. Here we introduce TARTE, a foundation model that transforms tables to knowledge-enhanced vector representations using the string to capture semantics. Pre-trained on large relational data, TARTE yields representations that facilitate subsequent learning with little additional cost. These representations can be fine-tuned or combined with other learners, giving models that push the state-of-the-art prediction performance and improve the prediction/computation performance trade-off. Specialized to a task or a domain, TARTE gives domain-specific representations that facilitate further learning. Our study demonstrates an effective approach to knowledge pre-training for tabular learning.