mlsanityData to 7 Oct 2026

Large Language Models to Enhance Bayesian Optimization

arXiv 2402.03921 ↗First posted 6 Feb 2024cs.LG cs.AIcited by 127

Not on the map. Why?

Cited by 24 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Bayesian optimization (BO) is a powerful approach for optimizing complex and expensive-to-evaluate black-box functions. Its importance is underscored in many applications, notably including hyperparameter tuning, but its efficacy depends on efficiently balancing exploration and exploitation. While there has been substantial progress in BO methods, striking this balance remains a delicate process. In this light, we present LLAMBO, a novel approach that integrates the capabilities of Large Language Models (LLM) within BO. At a high level, we frame the BO problem in natural language, enabling LLMs to iteratively propose and evaluate promising solutions conditioned on historical evaluations. More specifically, we explore how combining contextual understanding, few-shot learning proficiency, and domain knowledge of LLMs can improve model-based BO. Our findings illustrate that LLAMBO is effective at zero-shot warmstarting, and enhances surrogate modeling and candidate sampling, especially in the early stages of search when observations are sparse. Our approach is performed in context and does not require LLM finetuning. Additionally, it is modular by design, allowing individual components to be integrated into existing BO frameworks, or function cohesively as an end-to-end method. We empirically validate LLAMBO's efficacy on the problem of hyperparameter tuning, highlighting strong empirical performance across a range of diverse benchmarks, proprietary, and synthetic tasks.

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). Builds on 45 of its 78 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

45 of 78 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  11. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  12. 2023Large Language Models as Optimizerscited by 830
33 more
  1. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  2. 2018A Tutorial on Bayesian Optimizationcited by 734
  3. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 466
  4. 2015Deep Kernel Learningcited by 440
  5. 2023Larger language models do in-context learning differentlycited by 386
  6. 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 359
  7. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  8. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  9. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 229
  10. 2023Large Language Models as General Pattern Machinescited by 196
  11. 2018Neural Architecture Search with Bayesian Optimisation and Optimal Transportcited by 192
  12. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  13. 2023Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performancecited by 143
  14. 2022Evolution through Large Modelscited by 131
  15. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  16. 2023Language Model Crossover: Variation through Few-Shot Promptingcited by 119
  17. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  18. 2020HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisationcited by 81
  19. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  20. 2022HyperImpute: Generalized Iterative Imputation with Automatic Model Selectioncited by 81
  21. 2014Manifold Gaussian Processes for Regressioncited by 79
  22. 2019ChemBO: Bayesian Optimization of Small Organic Molecules with Synthesizable Recommendationscited by 77
  23. 2021HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenMLcited by 75
  24. 2021Few-Shot Bayesian Optimization with Deep Kernel Surrogatescited by 70
  25. 2023PFNs4BO: In-Context Learning for Bayesian Optimizationcited by 70
  26. 2023Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimescited by 52
  27. 2019Dataset2Vec: Learning Dataset Meta-Featurescited by 37
  28. 2023Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RLcited by 37
  29. 2022Sample-Then-Optimize Batch Neural Thompson Samplingcited by 25
  30. 2016Warm Starting Bayesian Optimizationcited by 21
  31. 2023Bayesian Optimization of Catalysis With In-Context Learningcited by 20
  32. 2022AutoPrognosis 2.0: Democratizing Diagnostic and Prognostic Modeling in Healthcare with Automated Machine Learningcited by 10
  33. 2018Efficient Global Optimization using Deep Gaussian Processescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 43; 2026: 56.

127 papers.

Sort:

By area:

  1. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  2. 2024LLM-SR: Scientific Equation Discovery via Programming with Large Language Modelscited by 117
  3. 2024LLMOPT: Learning to Define and Solve General Optimization Problems from Scratchcited by 102
  4. 2024AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMLcited by 84
  5. 2024A Systematic Survey on Large Language Models for Algorithm Designcited by 65
  6. 2024Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMscited by 62
  7. 2024BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experimentscited by 49
  8. 2024Autoformulation of Mathematical Optimization Models Using LLMscited by 48
  9. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 46
  10. 2025LLM-FE: Automated Feature Engineering for Tabular Data with LLMs as Evolutionary Optimizerscited by 35
  11. 2024LICO: Large Language Models for In-Context Molecular Optimizationcited by 26
  12. 2024Sequential Large Language Model-Based Hyper-parameter Optimizationcited by 23
38 more
  1. 2024REvolve: Reward Evolution with Large Language Models using Human Feedbackcited by 22
  2. 2024FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearchcited by 22
  3. 2024Amortized Probabilistic Conditioning for Optimization, Simulation and Inferencecited by 22
  4. 2024Position: Leverage Foundational Models for Black-Box Optimizationcited by 21
  5. 2025Optuna vs Code Llama: Are LLMs a New Paradigm for Hyperparameter Tuning?cited by 21
  6. 2024Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Surveycited by 19
  7. 2024Evolve Cost-aware Acquisition Functions Using Large Language Modelscited by 17
  8. 2024Bayesian Concept Bottleneck Models with LLM Priorscited by 16
  9. 2025AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Taskscited by 16
  10. 2025Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with LLMscited by 15
  11. 2025OptunaHub: A Platform for Black-Box Optimizationcited by 14
  12. 2024Automatically Learning Hybrid Digital Twins of Dynamical Systemscited by 13
  13. 2025Language-Based Bayesian Optimization Research Assistant (BORA)cited by 13
  14. 2025Planning of Heuristics: Strategic Planning on Large Language Models with Monte Carlo Tree Search for Automating Heuristic Optimizationcited by 13
  15. 2025LLINBO: Trustworthy LLM-in-the-Loop Bayesian Optimizationcited by 13
  16. 2025ORFS-agent: Tool-Using Agents for Chip Design Optimizationcited by 13
  17. 2026Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearchcited by 12
  18. 2024LLM-Enhanced Bayesian Optimization for Efficient Analog Layout Constraint Generationcited by 11
  19. 2025LLaMEA-BO: A Large Language Model Evolutionary Algorithm for Automatically Generating Bayesian Optimization Algorithmscited by 10
  20. 2025LLMs for Bayesian Optimization in Scientific Domains: Are We There Yet?cited by 10
  21. 2024Language Model Embeddings Can Be Sufficient for Bayesian Optimizationcited by 8
  22. 2024Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Taskscited by 8
  23. 2024Quantifying perturbation impacts for large language modelscited by 8
  24. 2025Reasoning BO: Enhancing Bayesian Optimization with Long-Context Reasoning Power of LLMscited by 8
  25. 2024Using Large Language Models for Parametric Shape Optimizationcited by 7
  26. 2025Large Language Model-Enhanced Multi-Armed Banditscited by 7
  27. 2025PABBO: Preferential Amortized Black-Box Optimizationcited by 7
  28. 2025LM-Searcher: Cross-domain Neural Architecture Search with LLMs via Unified Numerical Encodingcited by 7
  29. 2025Decision Tree Induction Through LLMs via Semantically-Aware Evolutioncited by 6
  30. 2025Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systemscited by 6
  31. 2025Improving LLM-based Global Optimization with Search Space Partitioningcited by 6
  32. 2025iDSE: Navigating Design Space Exploration in High-Level Synthesis Using LLMscited by 6
  33. 2025Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddingscited by 6
  34. 2025URSA: The Universal Research and Scientific Agentcited by 6
  35. 2026Adaptive Acquisition Selection for Bayesian Optimization with Large Language Modelscited by 6
  36. 2024Principled Bayesian Optimisation in Collaboration with Human Expertscited by 5
  37. 2025Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Modelscited by 5
  38. 2026MADE: Benchmark Environments for Closed-Loop Materials Discoverycited by 5

50 most cited of 127 All 127 (JSON)