# ScoringBench: A Benchmark for Evaluating Tabular Foundation Models with Proper Scoring Rules

arXiv: https://arxiv.org/abs/2603.29928 · HTML: https://mlsanity.com/paper/2603.29928/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 60 references that are in the map.
- First posted on arXiv 31 Mar 2026.

## Builds on (most cited first, 19 of 19)

- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [TabICLv2: A better, faster, scalable, and open tabular foundation model](https://mlsanity.com/paper/2602.11139/) (2026)
- [PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparison](https://mlsanity.com/paper/1703.00512/) (2017)
- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling](https://mlsanity.com/paper/2410.24210/) (2024)
- [Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data](https://mlsanity.com/paper/2407.04491/) (2024)
- [AMLB: an AutoML Benchmark](https://mlsanity.com/paper/2207.12560/) (2022)
- [TALENT: A Tabular Analytics and Learning Toolbox](https://mlsanity.com/paper/2407.04057/) (2024)
- [Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Data](https://mlsanity.com/paper/2507.03971/) (2025)
- [A Large-Scale Study of Probabilistic Calibration in Neural Network Regression](https://mlsanity.com/paper/2306.02738/) (2023)
- [Nonparametric Conditional Density Estimation in a High-Dimensional Regression Setting](https://mlsanity.com/paper/1604.00540/) (2016)
- [Proper scoring rules for estimation and forecast evaluation](https://mlsanity.com/paper/2504.01781/) (2025)
- [XGBoostLSS -- An extension of XGBoost to probabilistic forecasting](https://mlsanity.com/paper/1907.03178/) (2019)
- [A New Family of Divergences Originating from Model Adequacy Tests and Application to Robust Statistical Inference](https://mlsanity.com/paper/1611.05224/) (2016)
- [Distributional Regression with Tabular Foundation Models: Evaluating Probabilistic Predictions via Proper Scoring Rules](https://mlsanity.com/paper/2603.08206/) (2026)
- [Benchmarking Tabular Foundation Models for Conditional Density Estimation in Regression](https://mlsanity.com/paper/2603.26611/) (2026)
- [Joint Optimization of Neural Autoregressors via Scoring rules](https://mlsanity.com/paper/2601.05683/) (2026)

## Built on by (most cited first, 6 of 6)

- [Beyond IID: How General Are Tabular Foundation Models, Really?](https://mlsanity.com/paper/2606.30410/) (2026)
- [EXAONE Tabular 1.0 : Technical Report](https://mlsanity.com/paper/2608.25774/) (2026)
- [TabDPT-Turbo: Efficient In-Context Learning for Tabular Prediction](https://mlsanity.com/paper/2608.01400/) (2026)
- [Causilo Technical Report](https://mlsanity.com/paper/2609.22866/) (2026)
- [LoopICL: Looping a single transformer block to solve tabular tasks](https://mlsanity.com/paper/2609.36108/) (2026)
- [Characterizing the Separability Gap in Prior-Data Fitted Networks](https://mlsanity.com/paper/2610.14989/) (2026)

## Abstract (arXiv)

Tabular foundation models such as TabPFN, TabICL, ExaOneTabular, and Nori already produce full predictive distributions, yet prevailing regression benchmarks evaluate them almost exclusively via point-estimate metrics such as RMSE. This overlooks the distributional information these models are designed to provide. We introduce ScoringBench, an open and extensible benchmark that evaluates tabular regression models under a comprehensive suite of proper scoring rules - including (weighted) CRPS, energy scores and interval scores - alongside standard point metrics. ScoringBench covers 101 regression datasets from diverse domains, supports transparent community contributions via a git-based leaderboard, and provides two complementary ranking protocols: an ordinal Demsar/autorank approach and a magnitude-preserving z-score ranking approach. Our evaluation of various model types, including (fine-tuned) tabular foundation models, gradient-boosted trees, and MLPs, shows that model rankings shift depending on the chosen scoring rule. Models that perform well on point-estimate metrics can rank lower on probabilistic ones. These findings demonstrate that evaluation metric selection is a core modeling decision and a domain-specific requirement for applications where certain types of errors carry higher consequences.