# Position: The Future of Bayesian Prediction Is Prior-Fitted

arXiv: https://arxiv.org/abs/2505.23947 · HTML: https://mlsanity.com/paper/2505.23947/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 91 references that are in the map.
- First posted on arXiv 29 May 2025.

## Builds on (most cited first, 30 of 57)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [The frontier of simulation-based inference](https://mlsanity.com/paper/1911.01429/) (2019)
- [Reducing Transformer Depth on Demand with Structured Dropout](https://mlsanity.com/paper/1909.11556/) (2019)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [Neural Processes](https://mlsanity.com/paper/1807.01622/) (2018)
- [Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection](https://mlsanity.com/paper/2306.04637/) (2023)
- [TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentation](https://mlsanity.com/paper/2103.10158/) (2021)
- [When Do Neural Nets Outperform Boosted Trees on Tabular Data?](https://mlsanity.com/paper/2305.02997/) (2023)
- [Automatic Posterior Transformation for Likelihood-Free Inference](https://mlsanity.com/paper/1905.07488/) (2019)
- [End-to-end symbolic regression with transformers](https://mlsanity.com/paper/2204.10532/) (2022)
- [Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression](https://mlsanity.com/paper/2306.15063/) (2023)
- [Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modeling](https://mlsanity.com/paper/2207.04179/) (2022)
- [ForecastPFN: Synthetically-Trained Zero-Shot Forecasting](https://mlsanity.com/paper/2311.01933/) (2023)
- [Flow Matching for Scalable Simulation-Based Inference](https://mlsanity.com/paper/2305.17161/) (2023)
- [The Devil in Linear Transformer](https://mlsanity.com/paper/2210.10340/) (2022)
- [Transformers need glasses! Information over-squashing in language tasks](https://mlsanity.com/paper/2406.04267/) (2024)
- [Amortized Inference for Causal Structure Learning](https://mlsanity.com/paper/2205.12934/) (2022)
- [All-in-one simulation-based inference](https://mlsanity.com/paper/2404.09636/) (2024)
- [PFNs4BO: In-Context Learning for Bayesian Optimization](https://mlsanity.com/paper/2305.17535/) (2023)
- [Statistical Foundations of Prior-Data Fitted Networks](https://mlsanity.com/paper/2305.11097/) (2023)

## Built on by (most cited first, 20 of 20)

- [TabPFN-Wide: Continued Pre-Training for Extreme Feature Counts](https://mlsanity.com/paper/2510.06162/) (2025)
- [In-Context Learning of Temporal Point Processes with Foundation Inference Models](https://mlsanity.com/paper/2509.24762/) (2025)
- [Survival In-Context: Amortized Bayesian Survival Analysis via Prior-Fitted Networks](https://mlsanity.com/paper/2603.29475/) (2026)
- [Minimum Distance Summaries for Robust Neural Posterior Estimation](https://mlsanity.com/paper/2602.09161/) (2026)
- [SurvivalPFN: Amortizing Survival Prediction via In-Context Bayesian Inference](https://mlsanity.com/paper/2605.15488/) (2026)
- [Time-Aware Prior Fitted Networks for Zero-Shot Forecasting with Exogenous Variables](https://mlsanity.com/paper/2603.15802/) (2026)
- [Efficient Adaptive Data Acquisition via Pretrained Belief Representations](https://mlsanity.com/paper/2606.25197/) (2026)
- [PerturbPFN: Probing the Limits of Synthetic Priors in Drug Perturbation Modelling](https://mlsanity.com/paper/2607.23447/) (2026)
- [LLM Flow Processes for Text-Conditioned Regression](https://mlsanity.com/paper/2601.06147/) (2026)
- [Transformers Can Learn Posterior Predictive Distributions In-Context](https://mlsanity.com/paper/2605.26713/) (2026)
- [Multi-Task Bayesian In-Context Learning](https://mlsanity.com/paper/2606.20538/) (2026)
- [On Foundation Models for Temporal Point Processes to Accelerate Scientific Discovery](https://mlsanity.com/paper/2510.12640/) (2025)
- [Nearly Optimal Bayesian Inference for Structural Missingness](https://mlsanity.com/paper/2601.18500/) (2026)
- [Mechanistic Evidence for Spectral Structures in Prior-Data Fitted Networks](https://mlsanity.com/paper/2601.21731/) (2026)
- [Amortising Inference and Meta-Learning Priors in Neural Networks](https://mlsanity.com/paper/2602.08782/) (2026)
- [The Good, the Bad, and the Ugly of Markov Boundary for Tabular Prediction](https://mlsanity.com/paper/2605.29411/) (2026)
- [In-Context Learning for Latent Space Bayesian Optimization](https://mlsanity.com/paper/2606.09664/) (2026)
- [Reinforcement Learning Foundation Models Should Already Be A Thing](https://mlsanity.com/paper/2606.18812/) (2026)
- [TabPFN beyond Tabular Data: Calibration and Accuracy on Multimodal Embeddings](https://mlsanity.com/paper/2607.11007/) (2026)
- [An Investigation of the NeurIPS and ICML 2025 Position Tracks](https://mlsanity.com/paper/2608.16894/) (2026)

## Abstract (arXiv)

Training neural networks on randomly generated artificial datasets yields Bayesian models that capture the prior defined by the dataset-generating distribution. Prior-data Fitted Networks (PFNs) are a class of methods designed to leverage this insight. In an era of rapidly increasing computational resources for pre-training and a near stagnation in the generation of new real-world data in many applications, PFNs are poised to play a more important role across a wide range of applications. They enable the efficient allocation of pre-training compute to low-data scenarios. Originally applied to small Bayesian modeling tasks, the field of PFNs has significantly expanded to address more complex domains and larger datasets. This position paper argues that PFNs and other amortized inference approaches represent the future of Bayesian inference, leveraging amortized learning to tackle data-scarce problems. We thus believe they are a fruitful area of research. In this position paper, we explore their potential and directions to address their current limitations.