# The Illusion of Generalization in Tabular Language Models

arXiv: https://arxiv.org/abs/2602.04031 · HTML: https://mlsanity.com/paper/2602.04031/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Feb 2026.

## Built on by (most cited first, 4 of 4)

- [Beyond IID: How General Are Tabular Foundation Models, Really?](https://mlsanity.com/paper/2606.30410/) (2026)
- [STRABLE: Benchmarking Tabular Machine Learning with Strings](https://mlsanity.com/paper/2605.12292/) (2026)
- [MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image](https://mlsanity.com/paper/2605.10616/) (2026)
- [The Illusion of Replacement: Rethinking Specialized Machine Learning Models in the Foundation Model Era](https://mlsanity.com/paper/2608.28980/) (2026)

## Abstract (arXiv)

Tabular Language Models (TLMs) have been claimed to achieve strong generalization for tabular prediction. We conduct a systematic re-evaluation of Tabula-8B as a representative TLM, utilizing 165 datasets from the UniPredict benchmark. Our investigation reveals three findings. First, binary and categorical classification achieve near-zero median lift over majority-class baselines and strong aggregate performance is driven entirely by quartile classification tasks. Second, top-performing datasets exhibit pervasive contamination, including complete train-test overlap and task-level leakage that evades standard deduplication. Third, instruction-tuning without tabular exposure recovers 92.2% of standard classification performance and on quartile classification, format familiarity closes 71.3% of the gap with the residual attributable to contaminated datasets. These findings suggest claimed generalization likely reflects evaluation artifacts rather than learned tabular reasoning. We conclude with recommendations for strengthening TLM evaluation.