mlsanityData to 7 Oct 2026

Pretrained Optimization Model for Zero-Shot Black Box Optimization

arXiv 2405.03728 ↗First posted 6 May 2024cs.NE cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Zero-shot optimization involves optimizing a target task that was not seen during training, aiming to provide the optimal solution without or with minimal adjustments to the optimizer. It is crucial to ensure reliable and robust performance in various applications. Current optimizers often struggle with zero-shot optimization and require intricate hyperparameter tuning to adapt to new tasks. To address this, we propose a Pretrained Optimization Model (POM) that leverages knowledge gained from optimizing diverse tasks, offering efficient solutions to zero-shot optimization through direct application or fine-tuning with few-shot samples. Evaluation on the BBOB benchmark and two robot control tasks demonstrates that POM outperforms state-of-the-art black-box optimization methods, especially for high-dimensional tasks. Fine-tuning POM with a small number of samples and budget yields significant performance improvements. Moreover, POM demonstrates robust generalization across diverse task distributions, dimensions, population sizes, and optimization horizons. For code implementation, see https://github.com/ninja-wm/POM/.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 34 of its 61 references that are in the map. First posted on arXiv 6 May 2024.

Builds on

34 of 61 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  4. 2023Large Language Models as Optimizerscited by 830
  5. 2018Deep learning for time series classification: a reviewcited by 575
  6. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  7. 2016The CMA Evolution Strategy: A Tutorialcited by 364
  8. 2019Scalable Global Optimization via Local Bayesian Optimizationcited by 359
  9. 2024Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Modelcited by 258
  10. 2019Understanding and Improving Layer Normalizationcited by 213
  11. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 151
  12. 2017Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learningcited by 148
22 more
  1. 2016COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Settingcited by 135
  2. 2022Evolution through Large Modelscited by 131
  3. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  4. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 126
  5. 2023Language Model Crossover: Variation through Few-Shot Promptingcited by 119
  6. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 86
  7. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  8. 2023Diffusion Models for Black-Box Optimizationcited by 75
  9. 2023Algorithm Evolution Using Large Language Modelcited by 52
  10. 2024LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generationcited by 45
  11. 2022Discovering Evolution Strategies via Meta-Black-Box Optimizationcited by 44
  12. 2022Generative Pretraining for Black-Box Optimizationcited by 36
  13. 2023MetaBox: A Benchmark Platform for Meta-Black-Box Optimization with Reinforcement Learningcited by 35
  14. 2023Discovering Attention-Based Genetic Algorithms via Meta-Black-Box Optimizationcited by 33
  15. 2024Deep Reinforcement Learning for Dynamic Algorithm Selection: A Proof-of-Principle Study on Differential Evolutioncited by 24
  16. 2023ExPT: Synthetic Pretraining for Few-Shot Experimental Designcited by 22
  17. 2024Symbol: Generating Flexible Black-Box Optimizers through Symbolic Equation Learningcited by 22
  18. 2021Analogous to Evolutionary Algorithm: Designing a Unified Sequence Modelcited by 20
  19. 2024Exploring the True Potential: Evaluating the Black-box Optimization Capability of Large Language Modelscited by 19
  20. 2023Comparison of High-Dimensional Bayesian Optimization Algorithms on BBOBcited by 18
  21. 2017On the Relationship Between the OpenAI Evolution Strategy and Stochastic Gradient Descentcited by 16
  22. 2021Meta Learning Black-Box Population-Based Optimizerscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 4.

13 papers.

Sort:

  1. 2024Deep Insights into Automated Optimization with Large Language Models and Evolutionary Algorithmscited by 12
  2. 2025MetaBox-v2: A Unified Benchmark Platform for Meta-Black-Box Optimizationcited by 9
  3. 2025A Systematic Survey on Large Language Models for Evolutionary Optimization: From Modeling to Solvingcited by 9
  4. 2024EARL-BO: Reinforcement Learning for Multi-Step Lookahead, High-Dimensional Bayesian Optimizationcited by 6
  5. 2025Advancing CMA-ES with Learning-Based Cooperative Coevolution for Scalable Optimizationcited by 4
  6. 2025KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuningcited by 4
  7. 2026Task-free Adaptive Meta Black-box Optimizationcited by 2
  8. 2026From Heuristic Selection to Automated Algorithm Design: LLMs Benefit from Strong Priorscited by 2
  9. 2025Learning Evolution via Optimization Knowledge Adaptationcited by 1
  10. 2025ZeroShotOpt: Towards Zero-Shot Pretrained Models for Efficient Black-Box Optimizationcited by 1
  11. 2025Learning Where, What and How to Transfer: A Multi-Role Reinforcement Learning Approach for Evolutionary Multitaskingcited by 1
  12. 2026Detect and Act: Automated Dynamic Optimizer through Meta-Black-Box Optimizationcited by 1
1 more
  1. 2026Hyper Algorithm Design Agent: Evolving Learnable Optimizer from Zerocited by 0