# Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules

arXiv: https://arxiv.org/abs/2603.08206 · HTML: https://mlsanity.com/paper/2603.08206/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 71 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [TabICLv2: A better, faster, scalable, and open tabular foundation model](https://mlsanity.com/paper/2602.11139/) (2026)
- [LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence](https://mlsanity.com/paper/2509.03505/) (2025)
- [Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models](https://mlsanity.com/paper/2510.21204/) (2025)
- [TALENT: A Tabular Analytics and Learning Toolbox](https://mlsanity.com/paper/2407.04057/) (2024)
- [TabReD: Analyzing Pitfalls and Filling the Gaps in Tabular Deep Learning Benchmarks](https://mlsanity.com/paper/2406.19380/) (2024)
- [Regression Diagnostics meets Forecast Evaluation: Conditional Calibration, Reliability Diagrams, and Coefficient of Determination](https://mlsanity.com/paper/2108.03210/) (2021)
- [Isotonic Distributional Regression](https://mlsanity.com/paper/1909.03725/) (2019)
- [Forecaster's Dilemma: Extreme Events and Forecast Evaluation](https://mlsanity.com/paper/1512.09244/) (2015)
- [Nonparametric Conditional Density Estimation in a High-Dimensional Regression Setting](https://mlsanity.com/paper/1604.00540/) (2016)
- [From Aleatoric to Epistemic: Exploring Uncertainty Quantification Techniques in Artificial Intelligence](https://mlsanity.com/paper/2501.03282/) (2025)
- [Proper scoring rules for estimation and forecast evaluation](https://mlsanity.com/paper/2504.01781/) (2025)
- [TabTune: A Unified Library for Inference and Fine-Tuning Tabular Foundation Models](https://mlsanity.com/paper/2511.02802/) (2025)
- [In-sample calibration yields conformal calibration guarantees](https://mlsanity.com/paper/2503.03841/) (2025)
- [Beyond Strictly Proper Scoring Rules: The Importance of Being Local](https://mlsanity.com/paper/2012.12499/) (2020)
- [Decompositions of the mean continuous ranked probability score](https://mlsanity.com/paper/2311.14122/) (2023)
- [Proper Scoring Rules for Multivariate Probabilistic Forecasts based on Aggregation and Transformation](https://mlsanity.com/paper/2407.00650/) (2024)
- [Exploring Fine-Tuning for Tabular Foundation Models](https://mlsanity.com/paper/2601.09654/) (2026)
- [Reassessing How to Compare and Improve the Calibration of Machine Learning Models](https://mlsanity.com/paper/2406.04068/) (2024)
- [Enforcing tail calibration when training probabilistic forecast models](https://mlsanity.com/paper/2506.13687/) (2025)
- [Classifier Calibration at Scale: An Empirical Study of Model-Agnostic Post-Hoc Methods](https://mlsanity.com/paper/2601.19944/) (2026)
- [Joint Optimization of Neural Autoregressors via Scoring rules](https://mlsanity.com/paper/2601.05683/) (2026)

## Built on by (most cited first, 3 of 3)

- [Beyond IID: How General Are Tabular Foundation Models, Really?](https://mlsanity.com/paper/2606.30410/) (2026)
- [ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules](https://mlsanity.com/paper/2603.29928/) (2026)
- [Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression](https://mlsanity.com/paper/2603.26611/) (2026)

## Abstract (arXiv)

We study how the objective used to train or fine-tune a tabular foundation model affects its distributional predictions. Models such as TabPFN and TabICL provide forecasts of the full conditional distribution, but regression benchmarks including TabArena and TALENT assess them mainly through RMSE and R2. Such assessments reward accurate conditional means while leaving the remaining distributional information unassessed. We therefore examine proper scoring rules as objectives for training and fine-tuning and as criteria for evaluating and ranking these models. Every strictly proper rule favors the true distribution at the population level, however, this shared optimum does not determine how learning proceeds with a finite sample. Our examples illustrate score-dependent preferences between forecasts, and our fine-tuning experiments show differences in predictive performance under different objectives. Experiments that fine-tune TabPFN and TabICL on several datasets reveal that changing the training objective can change performance across evaluation scores, with non-trivial interactions between the two. By connecting established methods from distributional regression, scoring-rule theory, and foundation-model adaptation, we show that selecting the score is part of specifying the model and can measurably alter the distribution it learns.