# Do Tabular Foundation Models Agree with Themselves?

arXiv: https://arxiv.org/abs/2608.06004 · HTML: https://mlsanity.com/paper/2608.06004/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 26 references that are in the map.
- First posted on arXiv 6 Aug 2026.

## Builds on (most cited first, 12 of 12)

- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [TabDPT: Scaling Tabular Foundation Models on Real Data](https://mlsanity.com/paper/2410.18164/) (2024)
- [A Deep and Tractable Density Estimator](https://mlsanity.com/paper/1310.1757/) (2013)
- [Training and Inference on Any-Order Autoregressive Models the Right Way](https://mlsanity.com/paper/2205.13554/) (2022)
- [OpenML-Python: an extensible Python API for OpenML](https://mlsanity.com/paper/1911.02490/) (2019)
- [Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Models](https://mlsanity.com/paper/2504.17660/) (2025)
- [Guaranteed Deterministic Bounds on the Total Variation Distance between Univariate Mixtures](https://mlsanity.com/paper/1806.11311/) (2018)
- [TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction](https://mlsanity.com/paper/2608.01400/) (2026)
- [On the Conditioning Consistency Gap in Conditional Neural Processes](https://mlsanity.com/paper/2604.19312/) (2026)
- [Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepancies](https://mlsanity.com/paper/2505.08739/) (2025)

## Built on by (most cited first, 3 of 3)

- [Understanding the Surprising Generalization Properties of Tabular Foundation Models](https://mlsanity.com/paper/2608.17957/) (2026)
- [Tabular foundation models for non-tabular tasks](https://mlsanity.com/paper/2608.22594/) (2026)
- [From Synthetic Priors to Model Behavior: Structural Coverage in Tabular Foundation Models](https://mlsanity.com/paper/2609.06912/) (2026)

## Abstract (arXiv)

Tabular Foundation Models (TFMs) are currently the best approach to tabular prediction problems. They are constructed as transformers that approximate the Bayesian posterior predictive distribution based on a pre-training prior. These univariate predictors can be converted into multivariate ones autoregressively by sampling one target and adding it to the features. However, the faithfulness of the resulting joint has not been investigated. Furthermore, TFMs cannot be evaluated against the posterior itself, at least not on real-world datasets, because the ground-truth distribution is unknown. We therefore propose asking a different question: could a model's predictions result from any joint distribution? To answer this question, we pose two requirements that any such model must satisfy. The first is marginalization consistency, which demands that marginalized conditionals are equal to directly predicted marginals. The second is factorization consistency, which demands that different factorization orders result in equal joint distributions. Every TFM that we evaluate violates both of these requirements for both classification and regression across all datasets.