mlsanityData to 14 Oct 2026

How Well Do Prior-Data Fitted Networks Amortize Bayesian Inference?

arXiv 2610.13764 ↗First posted 9 Oct 2026stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Prior-data fitted networks (PFNs) are trained to approximate the Bayesian posterior predictive distribution (PPD) of a synthetic prior, yet results about the approximation quality are surprisingly thin. Taking an empirical risk minimization perspective, we decompose the deployment error of PFNs into prior, context, and amortization error, and measure the latter against a reference PPD under a realistic prior. Results show that PFNs generally recover the targeted PPD very well. Nevertheless, we find that parts of the residual amortization error persist, especially in textitrare settings under the prior pretraining distribution, even when scaling the model size, which manifests in disagreement between independently pretrained PFNs and leaves uncertainty about the predictor at inference time. As a remedy, we study multiple types of Bayesian model averages and find that these can consistently reduce this uncertainty and improve predictions up to full-size TabICLv2 models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 49 of its 75 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

49 of 75 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  5. 2019The frontier of simulation-based inferencecited by 502
  6. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
  7. 2021Transformers Can Do Bayesian Inferencecited by 360
  8. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 343
  9. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 289
  10. 2020Query-Key Normalization for Transformerscited by 280
  11. 2025TabArena: A Living Benchmark for Machine Learning on Tabular Datacited by 176
  12. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
37 more
  1. 2026TabICLv2: A better, faster, scalable, and open tabular foundation modelcited by 144
  2. 2025TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Modelscited by 132
  3. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 113
  4. 2021Repulsive Deep Ensembles are Bayesiancited by 104
  5. 2025LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligencecited by 79
  6. 2025Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Modelscited by 77
  7. 2023Statistical Foundations of Prior-Data Fitted Networkscited by 65
  8. 2024Variational Learning is Effective for Large Deep Networkscited by 65
  9. 2024Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AIcited by 62
  10. 2024Retrieval & Fine-Tuning for In-Context Tabular Modelscited by 59
  11. 2024Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspectivecited by 57
  12. 2013Understanding predictive information criteria for Bayesian modelscited by 52
  13. 2024TALENT: A Tabular Analytics and Learning Toolboxcited by 52
  14. 2025Do-PFN: In-Context Learning for Causal Effect Estimationcited by 51
  15. 2021Detecting Model Misspecification in Amortized Bayesian Inference with Neural Networkscited by 49
  16. 2025CausalPFN: Amortized Causal Effect Estimation via In-Context Learningcited by 49
  17. 2021Martingale posterior distributionscited by 45
  18. 2025Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Datacited by 41
  19. 2025TabPFN Unleashed: A Scalable and Effective Solution to Tabular Classification Problemscited by 36
  20. 2024Mixture of In-Context Prompters for Tabular PFNscited by 33
  21. 2024Fine-tuned In-Context Learning Transformers are Excellent Tabular Data Classifierscited by 24
  22. 2023Towards Efficient MCMC Sampling in Bayesian Neural Networks by Exploiting Symmetrycited by 21
  23. 2026Beyond IID: How General Are Tabular Foundation Models, Really?cited by 20
  24. 2025When can in-context learning generalize out of task distribution?cited by 16
  25. 2025nanoTabPFN: A Lightweight and Educational Reimplementation of TabPFNcited by 15
  26. 2026EXAONE Tabular 1.0 : Technical Reportcited by 10
  27. 2024Bayes' Power for Explaining In-Context Learning Generalizationscited by 8
  28. 2023Fluctuation without dissipation: Microcanonical Langevin Monte Carlocited by 7
  29. 2026Can Microcanonical Langevin Dynamics Leverage Mini-Batch Gradient Noise?cited by 7
  30. 2026Uncertainty Decomposition for Bayes-Filtered Transformers via Bayesian Predictive Inferencecited by 6
  31. 2026TabFM: A Zero-Shot Foundation Model for Tabular Datacited by 6
  32. 2025Uncertainty Quantification for Prior-Data Fitted Networks using Martingale Posteriorscited by 5
  33. 2026Towards Evaluating Data Priors for Tabular Foundation Modelscited by 4
  34. 2026On the Interplay of Priors and Overparametrization in Bayesian Neural Network Posteriorscited by 3
  35. 2026TabH2O: A Unified Foundation Model for Tabular Predictioncited by 3
  36. 2026Mind the Gap? A Distributional Comparison of Real and Synthetic Priors for Tabular Foundation Modelscited by 1
  37. 2026From Synthetic Priors to Model Behavior: Structural Coverage in Tabular Foundation Modelscited by 1

Built on by

Not cited yet.