# How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

arXiv: https://arxiv.org/abs/2610.13764 · HTML: https://mlsanity.com/paper/2610.13764/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 49 of its 75 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 30 of 49)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [The frontier of simulation-based inference](https://mlsanity.com/paper/1911.01429/) (2019)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1402.4102/) (2014)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [Query-Key Normalization for Transformers](https://mlsanity.com/paper/2010.04245/) (2020)
- [TabArena: A Living Benchmark for Machine Learning on Tabular Data](https://mlsanity.com/paper/2506.16791/) (2025)
- [Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression](https://mlsanity.com/paper/2306.15063/) (2023)
- [TabICLv2: A better, faster, scalable, and open tabular foundation model](https://mlsanity.com/paper/2602.11139/) (2026)
- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [TabDPT: Scaling Tabular Foundation Models on Real Data](https://mlsanity.com/paper/2410.18164/) (2024)
- [Repulsive Deep Ensembles are Bayesian](https://mlsanity.com/paper/2106.11642/) (2021)
- [LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence](https://mlsanity.com/paper/2509.03505/) (2025)
- [Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models](https://mlsanity.com/paper/2510.21204/) (2025)
- [Statistical Foundations of Prior-Data Fitted Networks](https://mlsanity.com/paper/2305.11097/) (2023)
- [Variational Learning is Effective for Large Deep Networks](https://mlsanity.com/paper/2402.17641/) (2024)
- [Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI](https://mlsanity.com/paper/2402.00809/) (2024)
- [Retrieval & Fine-Tuning for In-Context Tabular Models](https://mlsanity.com/paper/2406.05207/) (2024)
- [Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective](https://mlsanity.com/paper/2406.00793/) (2024)
- [Understanding predictive information criteria for Bayesian models](https://mlsanity.com/paper/1307.5928/) (2013)
- [TALENT: A Tabular Analytics and Learning Toolbox](https://mlsanity.com/paper/2407.04057/) (2024)
- [Do-PFN: In-Context Learning for Causal Effect Estimation](https://mlsanity.com/paper/2506.06039/) (2025)
- [Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networks](https://mlsanity.com/paper/2112.08866/) (2021)
- [CausalPFN: Amortized Causal Effect Estimation via In-Context Learning](https://mlsanity.com/paper/2506.07918/) (2025)
- [Martingale posterior distributions](https://mlsanity.com/paper/2103.15671/) (2021)
- [Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Data](https://mlsanity.com/paper/2507.03971/) (2025)

## Abstract (arXiv)

Prior-data fitted networks (PFNs) are trained to approximate the Bayesian posterior predictive distribution (PPD) of a synthetic prior, yet results about the approximation quality are surprisingly thin. Taking an empirical risk minimization perspective, we decompose the deployment error of PFNs into prior, context, and amortization error, and measure the latter against a reference PPD under a realistic prior. Results show that PFNs generally recover the targeted PPD very well. Nevertheless, we find that parts of the residual amortization error persist, especially in textitrare settings under the prior pretraining distribution, even when scaling the model size, which manifests in disagreement between independently pretrained PFNs and leaves uncertainty about the predictor at inference time. As a remedy, we study multiple types of Bayesian model averages and find that these can consistently reduce this uncertainty and improve predictions up to full-size TabICLv2 models.