mlsanityData to 14 Oct 2026

Position: The Future of Bayesian Prediction Is Prior-Fitted

arXiv 2505.23947 ↗First posted 29 May 2025cs.LG cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Training neural networks on randomly generated artificial datasets yields Bayesian models that capture the prior defined by the dataset-generating distribution. Prior-data Fitted Networks (PFNs) are a class of methods designed to leverage this insight. In an era of rapidly increasing computational resources for pre-training and a near stagnation in the generation of new real-world data in many applications, PFNs are poised to play a more important role across a wide range of applications. They enable the efficient allocation of pre-training compute to low-data scenarios. Originally applied to small Bayesian modeling tasks, the field of PFNs has significantly expanded to address more complex domains and larger datasets. This position paper argues that PFNs and other amortized inference approaches represent the future of Bayesian inference, leveraging amortized learning to tackle data-scarce problems. We thus believe they are a fruitful area of research. In this position paper, we explore their potential and directions to address their current limitations.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 57 of its 91 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

57 of 91 references.

References · 57

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  6. 2017Billion-scale similarity search with GPUscited by 2,497
  7. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  8. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  9. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  10. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  11. 2019The frontier of simulation-based inferencecited by 502
  12. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
38 more
  1. 2018Conditional Neural Processescited by 426
  2. 2021Transformers Can Do Bayesian Inferencecited by 360
  3. 2018Neural Processescited by 311
  4. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  5. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 227
  6. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
  7. 2019Automatic Posterior Transformation for Likelihood-Free Inferencecited by 219
  8. 2022End-to-end symbolic regression with transformerscited by 203
  9. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  10. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 127
  11. 2023ForecastPFN: Synthetically-Trained Zero-Shot Forecastingcited by 104
  12. 2023Flow Matching for Scalable Simulation-Based Inferencecited by 101
  13. 2022The Devil in Linear Transformercited by 98
  14. 2024Transformers need glasses! Information over-squashing in language taskscited by 91
  15. 2022Amortized Inference for Causal Structure Learningcited by 90
  16. 2024All-in-one simulation-based inferencecited by 74
  17. 2023PFNs4BO: In-Context Learning for Bayesian Optimizationcited by 73
  18. 2023Statistical Foundations of Prior-Data Fitted Networkscited by 65
  19. 2018Likelihood-free inference with emulator networkscited by 61
  20. 2024Retrieval & Fine-Tuning for In-Context Tabular Modelscited by 59
  21. 2024Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspectivecited by 57
  22. 2022Deep Symbolic Regression for Recurrent Sequencescited by 54
  23. 2024TuneTables: Context Optimization for Scalable Prior-Data Fitted Networkscited by 47
  24. 2023Tractable Function-Space Variational Inference in Bayesian Neural Networkscited by 41
  25. 2020Understanding Variational Inference in Function-Spacecited by 40
  26. 2020Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Modelscited by 39
  27. 2021Normalizing flows for atomic solidscited by 39
  28. 2023Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networkscited by 37
  29. 2024Mixture of In-Context Prompters for Tabular PFNscited by 33
  30. 2024Algorithmic progress in language modelscited by 31
  31. 2025Can Transformers Learn Full Bayesian Inference in Context?cited by 31
  32. 2024In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimizationcited by 26
  33. 2024Drift-Resilient TabPFN: In-Context Learning Temporal Distribution Shifts on Tabular Datacited by 25
  34. 2023Neuralizer: General Neuroimage Analysis without Re-Trainingcited by 24
  35. 2024When Can Transformers Count to n?cited by 24
  36. 2019Neural Density Estimation and Likelihood-free Inferencecited by 23
  37. 2020When Ensembling Smaller Models is More Efficient than Single Large Modelscited by 20
  38. 2024Mamba4Cast: Efficient Zero-Shot Time Series Forecasting with State Space Modelscited by 17

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 17.

20 papers.

Sort:

By area:

  1. 2025TabPFN-Wide: Continued Pre-Training for Extreme Feature Countscited by 16
  2. 2025In-Context Learning of Temporal Point Processes with Foundation Inference Modelscited by 10
  3. 2026Survival In-Context: Amortized Bayesian Survival Analysis via Prior-Fitted Networkscited by 6
  4. 2026Minimum Distance Summaries for Robust Neural Posterior Estimationcited by 3
  5. 2026SurvivalPFN: Amortizing Survival Prediction via In-Context Bayesian Inferencecited by 3
  6. 2026Time-Aware Prior Fitted Networks for Zero-Shot Forecasting with Exogenous Variablescited by 2
  7. 2026Efficient Adaptive Data Acquisition via Pretrained Belief Representationscited by 2
  8. 2026PerturbPFN: Probing the Limits of Synthetic Priors in Drug Perturbation Modellingcited by 2
  9. 2026LLM Flow Processes for Text-Conditioned Regressioncited by 1
  10. 2026Transformers Can Learn Posterior Predictive Distributions In-Contextcited by 1
  11. 2026Multi-Task Bayesian In-Context Learningcited by 1
  12. 2025On Foundation Models for Temporal Point Processes to Accelerate Scientific Discoverycited by 0
8 more
  1. 2026Nearly Optimal Bayesian Inference for Structural Missingnesscited by 0
  2. 2026Mechanistic Evidence for Spectral Structures in Prior-Data Fitted Networkscited by 0
  3. 2026Amortising Inference and Meta-Learning Priors in Neural Networkscited by 0
  4. 2026The Good, the Bad, and the Ugly of Markov Boundary for Tabular Predictioncited by 0
  5. 2026In-Context Learning for Latent Space Bayesian Optimizationcited by 0
  6. 2026Reinforcement Learning Foundation Models Should Already Be A Thingcited by 0
  7. 2026TabPFN beyond Tabular Data: Calibration and Accuracy on Multimodal Embeddingscited by 0
  8. 2026An Investigation of the NeurIPS and ICML 2025 Position Trackscited by 0