mlsanityData to 7 Oct 2026

FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch

arXiv 2406.04824 ↗First posted 7 Jun 2024cs.LG stat.MLcited by 22

Not on the map. Why?

Abstract (arXiv)

The sample efficiency of Bayesian optimization algorithms depends on carefully crafted acquisition functions (AFs) guiding the sequential collection of function evaluations. The best-performing AF can vary significantly across optimization problems, often requiring ad-hoc and problem-specific choices. This work tackles the challenge of designing novel AFs that perform well across a variety of experimental settings. Based on FunSearch, a recent work using Large Language Models (LLMs) for discovery in mathematical sciences, we propose FunBO, an LLM-based method that can be used to learn new AFs written in computer code by leveraging access to a limited number of evaluations for a set of objective functions. We provide the analytic expression of all discovered AFs and evaluate them on various global optimization benchmarks and hyperparameter optimization tasks. We show how FunBO identifies AFs that generalize well in and out of the training distribution of functions, thus outperforming established general-purpose AFs and achieving competitive performance against AFs that are customized to specific function types and are learned via transfer-learning algorithms.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 36 of its 64 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

36 of 64 references.

References · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023PaLM 2 Technical Reportcited by 1,154
  3. 2023Large Language Models as Optimizerscited by 830
  4. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  5. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  6. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
  7. 2024Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Modelcited by 258
  8. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  9. 2017Max-value Entropy Search for Efficient Bayesian Optimizationcited by 202
  10. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  11. 2022Evolution through Large Modelscited by 131
  12. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
24 more
  1. 2024Large Language Models to Enhance Bayesian Optimizationcited by 127
  2. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
  3. 2023Language Model Crossover: Variation through Few-Shot Promptingcited by 119
  4. 2022Thor: Wielding Hammers to Integrate Language Models and Automated Theorem Proverscited by 97
  5. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 86
  6. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  7. 2019ChemBO: Bayesian Optimization of Small Organic Molecules with Synthesizable Recommendationscited by 77
  8. 2023Can GPT-4 Perform Neural Architecture Search?cited by 74
  9. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  10. 2019Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimizationcited by 70
  11. 2021Few-Shot Bayesian Optimization with Deep Kernel Surrogatescited by 70
  12. 2023PFNs4BO: In-Context Learning for Bayesian Optimizationcited by 70
  13. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68
  14. 2019Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learningcited by 50
  15. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 46
  16. 2023Cost-Effective Hyperparameter Optimization for Large Language Model Generation Inferencecited by 40
  17. 2023End-to-End Meta-Bayesian Optimisation with Transformer Neural Processescited by 31
  18. 2021Reinforced Few-Shot Acquisition Function Learning for Bayesian Optimizationcited by 22
  19. 2024Position: Leverage Foundational Models for Black-Box Optimizationcited by 21
  20. 2023Hyperparameter Optimization for Large Language Model Instruction-Tuningcited by 17
  21. 2024Evolve Cost-aware Acquisition Functions Using Large Language Modelscited by 17
  22. 2024Reinforced In-Context Black-Box Optimizationcited by 14
  23. 2021Syndicated Bandits: A Framework for Auto Tuning Hyper-parameters in Contextual Bandit Algorithmscited by 10
  24. 2023Self-Adjusting Weighted Expected Improvement for Bayesian Optimizationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 10.

22 papers.

Sort:

By area:

  1. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
  2. 2024A Systematic Survey on Large Language Models for Algorithm Designcited by 65
  3. 2025Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMscited by 15
  4. 2025VLMgineer: Vision Language Models as Robotic Toolsmithscited by 9
  5. 2025PABBO: Preferential Amortized Black-Box Optimizationcited by 7
  6. 2025Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systemscited by 6
  7. 2025(X)-evolve: Solution space evolution powered by large language modelscited by 6
  8. 2026Closed-Loop LLM Discovery of Non-Standard Channel Priors in Vision Modelscited by 6
  9. 2026Adaptive Acquisition Selection for Bayesian Optimization with Large Language Modelscited by 6
  10. 2025LLM-Guided Search for Deletion-Correcting Codescited by 4
  11. 2025Discovering Heuristics with Large Language Models (LLMs) for Mixed-Integer Programs: Single-Machine Schedulingcited by 4
  12. 2025Large Language Model-Driven Surrogate-Assisted Evolutionary Algorithm for Expensive Optimizationcited by 2
10 more
  1. 2025Generative Bayesian Optimization: Generative Models as Acquisition Functionscited by 2
  2. 2025EvoVLMA: Evolutionary Vision-Language Model Adaptationcited by 1
  3. 2026LLM-Assisted Logic Rule Learning: Scaling Human Expertise for Time Series Anomaly Detectioncited by 1
  4. 2026Advancing Automated Algorithm Design via Evolutionary Stagewise Design with LLMscited by 1
  5. 2026Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimizationcited by 1
  6. 2026An Open-Source Training Dataset for Foundation Models for Black-box Optimizationcited by 0
  7. 2026The Kalman Evolve: Closing the Gap in Kalman Filtering via Interpretable Algorithm Discoverycited by 0
  8. 2026Evidence-Gated LLM Priors for Multi-Objective Bayesian Optimizationcited by 0
  9. 2026Algorithmic algorithm development with LLMs: A Case Study on LLM-Usage for Contraction Order Optimization in Tensor Networkscited by 0
  10. 2026Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasonerscited by 0