mlsanityData to 7 Oct 2026

Position: Leverage Foundational Models for Black-Box Optimization

arXiv 2405.03547 ↗First posted 6 May 2024cs.LG cs.AI cs.NEcited by 21

Not on the map. Why?

Abstract (arXiv)

Undeniably, Large Language Models (LLMs) have stirred an extraordinary wave of innovation in the machine learning research domain, resulting in substantial impact across diverse fields such as reinforcement learning, robotics, and computer vision. Their incorporation has been rapid and transformative, marking a significant paradigm shift in the field of machine learning research. However, the field of experimental design, grounded on black-box optimization, has been much less affected by such a paradigm shift, even though integrating LLMs with optimization presents a unique landscape ripe for exploration. In this position paper, we frame the field of black-box optimization around sequence-based foundation models and organize their relationship with previous literature. We discuss the most promising ways foundational language models can revolutionize optimization, which include harnessing the vast wealth of information encapsulated in free-form text to enrich task comprehension, utilizing highly flexible sequence models such as Transformers to engineer superior optimization strategies, and enhancing performance prediction over previously unseen search spaces.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 67 of its 94 references that are in the map. First posted on arXiv 6 May 2024.

Builds on

67 of 94 references.

References · 67

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
38 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  4. 2023PaLM 2 Technical Reportcited by 1,154
  5. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  6. 2016Neural Combinatorial Optimization with Reinforcement Learningcited by 600
  7. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  8. 2016The CMA Evolution Strategy: A Tutorialcited by 364
  9. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  10. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  11. 2021Transformers Can Do Bayesian Inferencecited by 338
  12. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  13. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  14. 2020Neural Architecture Search without Trainingcited by 241
  15. 2021Learning to Optimize: A Primer and A Benchmarkcited by 206
  16. 2016Learning to Optimizecited by 195
  17. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  18. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 137
  19. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  20. 2022Evolution through Large Modelscited by 131
  21. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  22. 2024Large Language Models to Enhance Bayesian Optimizationcited by 127
  23. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 126
  24. 2023Language Model Crossover: Variation through Few-Shot Promptingcited by 119
  25. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  26. 2020Predicting Neural Network Accuracy from Weightscited by 107
  27. 2022Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimizationcited by 96
  28. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 86
  29. 2023Batch Prompting: Efficient Inference with Large Language Model APIscited by 84
  30. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  31. 2019Combinatorial Bayesian Optimization using the Graph Cartesian Productcited by 78
  32. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  33. 2023A Survey of Large Language Models Attributioncited by 71
  34. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  35. 2019Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimizationcited by 70
  36. 2021Few-Shot Bayesian Optimization with Deep Kernel Surrogatescited by 70
  37. 2023PFNs4BO: In-Context Learning for Bayesian Optimizationcited by 70
  38. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 7; 2026: 9.

21 papers.

Sort:

By area:

  1. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  2. 2025Towards Robust Agentic CUDA Kernel Benchmarking, Verification, and Optimizationcited by 52
  3. 2024Toward Automated Algorithm Design: A Survey and Practical Guide to Meta-Black-Box-Optimizationcited by 23
  4. 2024FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearchcited by 22
  5. 2025LLINBO: Trustworthy LLM-in-the-Loop Bayesian Optimizationcited by 13
  6. 2024Language Model Embeddings Can Be Sufficient for Bayesian Optimizationcited by 8
  7. 2024Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Taskscited by 8
  8. 2025Decision Tree Induction Through LLMs via Semantically-Aware Evolutioncited by 6
  9. 2025Improving LLM-based Global Optimization with Search Space Partitioningcited by 6
  10. 2025Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddingscited by 6
  11. 2025How Should We Meta-Learn Reinforcement Learning Algorithms?cited by 5
  12. 2026Agentic Bayesian Optimization through Surrogate-Augmented Autoresearchcited by 5
9 more
  1. 2025Overtuning in Hyperparameter Optimizationcited by 4
  2. 2026Multi-Objective Hierarchical Optimization with Large Language Modelscited by 1
  3. 2026Diffusion Large Language Models for Black-Box Optimizationcited by 1
  4. 2026On the use of LLMs to generate a dataset of Neural Networkscited by 0
  5. 2026Multi-Agent LLMs for Adaptive Acquisition in Bayesian Optimizationcited by 0
  6. 2026An Open-Source Training Dataset for Foundation Models for Black-box Optimizationcited by 0
  7. 2026Agentic Calibration of Grey-Box Simulation Models: An LLM-Driven Alternativecited by 0
  8. 2026BBOWP-Bench: Evaluating LLMs on Black-Box Optimization Word Problemscited by 0
  9. 2026Agentic High-Dimensional Bayesian Optimization with Hypothesis- and Evidence-Guided Searchcited by 0