mlsanityData to 7 Oct 2026

PFNs4BO: In-Context Learning for Bayesian Optimization

arXiv 2305.17535 ↗First posted 27 May 2023cs.LG stat.MLcited by 70

Not on the map. Why?

Abstract (arXiv)

In this paper, we use Prior-data Fitted Networks (PFNs) as a flexible surrogate for Bayesian Optimization (BO). PFNs are neural processes that are trained to approximate the posterior predictive distribution (PPD) through in-context learning on any prior distribution that can be efficiently sampled from. We describe how this flexibility can be exploited for surrogate modeling in BO. We use PFNs to mimic a naive Gaussian process (GP), an advanced GP, and a Bayesian Neural Network (BNN). In addition, we show how to incorporate further information into the prior, such as allowing hints about the position of optima (user priors), ignoring irrelevant dimensions, and performing non-myopic BO by learning the acquisition function. The flexibility underlying these extensions opens up vast possibilities for using PFNs for BO. We demonstrate the usefulness of PFNs for BO in a large-scale evaluation on artificial GP samples and three different hyperparameter optimization testbeds: HPO-B, Bayesmark, and PD1. We publish code alongside trained models at github.com/automl/PFNs4BO.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 46 of its 71 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

46 of 71 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  7. 2018A Tutorial on Bayesian Optimizationcited by 734
  8. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  9. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  10. 2018Conditional Neural Processescited by 421
  11. 2014Stochastic Gradient Hamiltonian Monte Carlocited by 341
  12. 2021Transformers Can Do Bayesian Inferencecited by 338
34 more
  1. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  2. 2013Bayesian Optimization in a Billion Dimensions via Random Embeddingscited by 178
  3. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 146
  4. 2021High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspacescited by 139
  5. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 126
  6. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  7. 2017Constrained Bayesian Optimization with Noisy Experimentscited by 103
  8. 2019Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representationcited by 84
  9. 2020HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisationcited by 81
  10. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  11. 2014Manifold Gaussian Processes for Regressioncited by 79
  12. 2021HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenMLcited by 75
  13. 2016The Parallel Knowledge Gradient Method for Batch Bayesian Optimizationcited by 73
  14. 2021Few-Shot Bayesian Optimization with Deep Kernel Surrogatescited by 70
  15. 2022πBO: Augmenting Acquisition Functions with User Beliefs for Bayesian Optimizationcited by 69
  16. 2014Input Warping for Bayesian Optimization of Non-stationary Functionscited by 66
  17. 2021Pre-trained Gaussian Processes for Bayesian Optimizationcited by 63
  18. 2017Hyperparameter Importance Across Datasetscited by 62
  19. 2017Bayesian Optimization with Gradientscited by 59
  20. 2022Constrained Efficient Global Optimization of Expensive Black-box Functionscited by 49
  21. 2014Student-t Processes as Alternatives to Gaussian Processescited by 45
  22. 2022Bayesian Optimization over Discrete and Mixed Spaces via Probabilistic Reparameterizationcited by 35
  23. 2018Practical Transfer Learning for Bayesian Optimizationcited by 26
  24. 2020Incorporating Expert Prior in Bayesian Optimisation via Space Warpingcited by 23
  25. 2014Raiders of the Lost Architecture: Kernels for Bayesian Optimization in Conditional Parameter Spacescited by 16
  26. 2019Achieving Robustness to Aleatoric Uncertainty with Heteroscedastic Bayesian Optimisationcited by 11
  27. 2014Heteroscedastic Treed Bayesian Optimisationcited by 9
  28. 2020Neural Architecture Optimization with Graph VAEcited by 9
  29. 2019pySOT and POAP: An event-driven asynchronous framework for surrogate optimizationcited by 6
  30. 2020Neural Model-based Optimization with Right-Censored Observationscited by 6
  31. 2016Advancing Bayesian Optimization: The Mixed-Global-Local (MGL) Kernel and Length-Scale Cool Downcited by 5
  32. 2018Efficient Global Optimization using Deep Gaussian Processescited by 4
  33. 2022Sample-Efficient Optimisation with Probabilistic Transformer Surrogatescited by 3
  34. 2017Neural Networks for Predicting Algorithm Runtime Distributionscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 17; 2025: 24; 2026: 22.

70 papers.

Sort:

By area:

  1. 2024Large Language Models to Enhance Bayesian Optimizationcited by 127
  2. 2023ForecastPFN: Synthetically-Trained Zero-Shot Forecastingcited by 102
  3. 2024TuneTables: Context Optimization for Scalable Prior-Data Fitted Networkscited by 47
  4. 2025Do-PFN: In-Context Learning for Causal Effect Estimationcited by 46
  5. 2024Is Mamba Capable of In-Context Learning?cited by 45
  6. 2023Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networkscited by 37
  7. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  8. 2024In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimizationcited by 26
  9. 2023A General Framework for User-Guided Bayesian Optimizationcited by 22
  10. 2024FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearchcited by 22
  11. 2024Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyondcited by 22
  12. 2024Amortized Probabilistic Conditioning for Optimization, Simulation and Inferencecited by 22
38 more
  1. 2024Position: Leverage Foundational Models for Black-Box Optimizationcited by 21
  2. 2024Amortized Bayesian Experimental Design for Decision-Makingcited by 20
  3. 2025Position: The Future of Bayesian Prediction Is Prior-Fittedcited by 19
  4. 2024Reinforced In-Context Black-Box Optimizationcited by 14
  5. 2025OptunaHub: A Platform for Black-Box Optimizationcited by 14
  6. 2024Language Model Embeddings Can Be Sufficient for Bayesian Optimizationcited by 8
  7. 2024IM-Context: In-Context Learning for Imbalanced Regression Taskscited by 7
  8. 2024Monte Carlo Tree Search based Space Transfer for Black-box Optimizationcited by 7
  9. 2025PABBO: Preferential Amortized Black-Box Optimizationcited by 7
  10. 2025Simplifying Bayesian Optimization Via In-Context Direct Optimum Samplingcited by 7
  11. 2024Bayes' Power for Explaining In-Context Learning Generalizationscited by 6
  12. 2025Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddingscited by 6
  13. 2023Pseudo-Bayesian Optimizationcited by 4
  14. 2025Put CASH on Bandits: A Max K-Armed Problem for Automated Machine Learningcited by 4
  15. 2025Efficient Autoregressive Inference for Transformer Probabilistic Modelscited by 4
  16. 2024System-Aware Neural ODE Processes for Few-Shot Bayesian Optimizationcited by 3
  17. 2025EquiTabPFN: A Target-Permutation Equivariant Prior Fitted Networkscited by 3
  18. 2025None To Optima in Few Shots: Bayesian Optimization with MDP Priorscited by 3
  19. 2026MapPFN: Learning Causal Perturbation Maps in Contextcited by 3
  20. 2023MALIBO: Meta-learning for Likelihood-free Bayesian Optimizationcited by 2
  21. 2024Cost-Sensitive Multi-Fidelity Bayesian Optimization with Transfer of Learning Curve Extrapolationcited by 2
  22. 2025FoMEMO: Towards Foundation Models for Expensive Multi-objective Optimizationcited by 2
  23. 2025Cost-Sensitive Freeze-thaw Bayesian Optimization for Efficient Hyperparameter Tuningcited by 2
  24. 2025Informed Initialization for Bayesian Optimization and Active Learningcited by 2
  25. 2025Dynamic Priors in Bayesian Optimization for Hyperparameter Optimizationcited by 2
  26. 2025Uncertainty Quantification and Data Efficiency in AI: An Information-Theoretic Perspectivecited by 2
  27. 2026Automatic Configuration of LLM Post-Training Pipelinescited by 2
  28. 2026Efficient Adaptive Data Acquisition via Pretrained Belief Representationscited by 2
  29. 2026Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Modelscited by 2
  30. 2023Data-driven Prior Learning for Bayesian Optimisationcited by 1
  31. 2024GAMformer: Bridging Tabular Foundation Models and Interpretable Machine Learningcited by 1
  32. 2025Meta-Statistical Learning: Supervised Learning of Statistical Estimatorscited by 1
  33. 2025ABG-NAS: Adaptive Bayesian Genetic Neural Architecture Search for Graph Representation Learningcited by 1
  34. 2025ZeroShotOpt: Towards Zero-Shot Pretrained Models for Efficient Black-Box Optimizationcited by 1
  35. 2025SemanticOpt: Towards LLM-Based Semantic Black-Box Optimizationcited by 1
  36. 2025MIST: Mutual Information Estimation Via Supervised Trainingcited by 1
  37. 2025Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learningcited by 1
  38. 2026In-Context Black-Box Optimization with Unreliable Feedbackcited by 1

50 most cited of 70 All 70 (JSON)