mlsanityData to 14 Oct 2026

Language Model Embeddings Can Be Sufficient for Bayesian Optimization

arXiv 2410.10190 ↗First posted 14 Oct 2024cs.LG cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

Bayesian Optimization is ubiquitous in experimental design and black-box optimization for improving search efficiency. However, most existing approaches rely on regression models which are limited to fixed search spaces and structured, tabular input features. This paper explores the use of LLM embeddings over string inputs for in-context regression in Bayesian Optimization. Our results show that representing inputs as strings enables general-purpose regression across diverse domains, including synthetic, combinatorial, and hyperparameter optimization. Furthermore, our approach achieves optimization performance comparable to state-of-the-art Gaussian Process-based methods such as Google Vizier, and demonstrates potential for broader and more flexible applications.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 41 of its 50 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

41 of 50 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  8. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  10. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  11. 2020Efficient Transformers: A Surveycited by 790
  12. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
29 more
  1. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 551
  2. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  3. 2021Transformers Can Do Bayesian Inferencecited by 360
  4. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  5. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  6. 2019BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Searchcited by 179
  7. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 143
  8. 2024Large Language Models to Enhance Bayesian Optimizationcited by 135
  9. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 127
  10. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  11. 2022Local Latent Space Bayesian Optimization over Structured Inputscited by 79
  12. 2023PFNs4BO: In-Context Learning for Bayesian Optimizationcited by 73
  13. 2020Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarkscited by 67
  14. 2021Pre-trained Gaussian Processes for Bayesian Optimizationcited by 63
  15. 2021Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantificationcited by 57
  16. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 57
  17. 2024A Sober Look at LLMs for Material Discovery: Are They Actually Good for Bayesian Optimization Over Molecules?cited by 48
  18. 2024OmniPred: Language Models as Universal Regressorscited by 34
  19. 2024LICO: Large Language Models for In-Context Molecular Optimizationcited by 26
  20. 2023Bayesian Optimization over High-Dimensional Combinatorial Spaces via Dictionary-based Embeddingscited by 24
  21. 2024Localized Zeroth-Order Prompt Optimizationcited by 23
  22. 2023ExPT: Synthetic Pretraining for Few-Shot Experimental Designcited by 22
  23. 2024Position: Leverage Foundational Models for Black-Box Optimizationcited by 21
  24. 2021PyGlove: Symbolic Programming for Automated Machine Learningcited by 19
  25. 2021Bayesian Optimization over Permutation Spacescited by 17
  26. 2024The Vizier Gaussian Process Bandit Algorithmcited by 14
  27. 2019COCO: The Large Scale Black-Box Optimization Benchmarking (bbob-largescale) Test Suitecited by 12
  28. 2023Transfer Learning for Bayesian Optimization on Heterogeneous Search Spacescited by 9
  29. 2021Batch Bayesian Optimization on Permutations using the Acquisition Weighted Kernelcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 2.

8 papers.

  1. 2024Understanding LLM Embeddings for Regressioncited by 19
  2. 2025Large language models as uncertainty-calibrated optimizers for experimental discoverycited by 13
  3. 2025Decoding-based Regressioncited by 12
  4. 2025Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddingscited by 7
  5. 2025Reasoning-Intensive Regressioncited by 2
  6. 2026A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Searchcited by 2
  7. 2025Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learningcited by 1
  8. 2026Agentic High-Dimensional Bayesian Optimization with Hypothesis- and Evidence-Guided Searchcited by 0