mlsanityData to 14 Oct 2026

Towards Learning Universal Hyperparameter Optimizers with Transformers

arXiv 2205.13320 ↗First posted 26 May 2022cs.LG cs.AI stat.MLcited by 81

Not on the map. Why?

Abstract (arXiv)

Meta-learning hyperparameter optimization (HPO) algorithms from prior experiments is a promising approach to improve optimization efficiency over objective functions from a similar distribution. However, existing methods are restricted to learning from experiments sharing the same set of hyperparameters. In this paper, we introduce the OptFormer, the first text-based Transformer HPO framework that provides a universal end-to-end interface for jointly learning policy and function prediction when trained on vast tuning data from the wild, such as Google's Vizier database, one of the world's largest HPO datasets. Our extensive experiments demonstrate that the OptFormer can simultaneously imitate at least 7 different HPO algorithms, which can be further improved via its function uncertainty estimates. Compared to a Gaussian Process, the OptFormer also learns a robust prior distribution for hyperparameter response functions, and can thereby provide more accurate and better calibrated predictions. This work paves the path to future extensions for training a Transformer-based model as a general HPO optimizer.

Key facts. Cited by 81 papers in the map (data through 14 Oct 2026). Builds on 43 of its 77 references that are in the map. First posted on arXiv 26 May 2022.

Builds on

43 of 77 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  9. 2020Rethinking Attention with Performerscited by 1,601
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  12. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
31 more
  1. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  2. 2022A Generalist Agentcited by 808
  3. 2014OpenML: networked science in machine learningcited by 526
  4. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  5. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 416
  6. 2018Tune: A Research Platform for Distributed Model Selection and Trainingcited by 395
  7. 2016Learning to reinforcement learncited by 377
  8. 2021Transformers Can Do Bayesian Inferencecited by 360
  9. 2019Deep Learning for Symbolic Mathematicscited by 270
  10. 2022Online Decision Transformercited by 196
  11. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 148
  12. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 125
  13. 2021SymbolicGPT: A Generative Transformer Model for Symbolic Regressioncited by 105
  14. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  15. 2020PACOH: Bayes-Optimal Meta-Learning with PAC-Guaranteescited by 85
  16. 2016Multi-Information Source Optimizationcited by 84
  17. 2021HPO-B: A Large-Scale Reproducible Benchmark for Black-Box HPO based on OpenMLcited by 75
  18. 2021Few-Shot Bayesian Optimization with Deep Kernel Surrogatescited by 71
  19. 2019Meta-Learning Acquisition Functions for Transfer Learning in Bayesian Optimizationcited by 70
  20. 2021Pre-trained Gaussian Processes for Bayesian Optimizationcited by 63
  21. 2021Linear algebra with transformerscited by 63
  22. 2019Gradientless Descent: High-Dimensional Zeroth-Order Optimizationcited by 43
  23. 2021Regularized Behavior Value Estimationcited by 32
  24. 2018Regret bounds for meta Bayesian optimization with an unknown Gaussian process priorcited by 31
  25. 2016Warm Starting Bayesian Optimizationcited by 21
  26. 2021PyGlove: Symbolic Programming for Automated Machine Learningcited by 19
  27. 2019COCO: The Large Scale Black-Box Optimization Benchmarking (bbob-largescale) Test Suitecited by 12
  28. 2021Frugal Machine Learningcited by 11
  29. 2021Analyzing the Nuances of Transformers' Polynomial Simplification Abilitiescited by 9
  30. 2017JSON: data model, query languages and schema specificationcited by 7
  31. 2021Privileged Zero-Shot AutoMLcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 14; 2024: 28; 2025: 18; 2026: 17.

81 papers.

Sort:

By area:

  1. 2023Large Language Models as Optimizerscited by 864
  2. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  3. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  4. 2024Large Language Models to Enhance Bayesian Optimizationcited by 135
  5. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 131
  6. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
  7. 2023PFNs4BO: In-Context Learning for Bayesian Optimizationcited by 73
  8. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  9. 2023MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Taskscited by 66
  10. 2024Large Language Model Agent for Hyper-Parameter Optimizationcited by 64
  11. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 63
  12. 2024Is Mamba Capable of In-Context Learning?cited by 45
38 more
  1. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 35
  2. 2024OmniPred: Language Models as Universal Regressorscited by 34
  3. 2023Discovering Attention-Based Genetic Algorithms via Meta-Black-Box Optimizationcited by 33
  4. 2023End-to-End Meta-Bayesian Optimisation with Transformer Neural Processescited by 32
  5. 2024AutoGluon-Multimodal (AutoMM): Supercharging Multimodal AutoML with Foundation Modelscited by 30
  6. 2023Large Language Model for Multi-objective Evolutionary Optimizationcited by 29
  7. 2024Large Language Model-Based Evolutionary Optimizer: Reasoning with elitismcited by 27
  8. 2024In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimizationcited by 26
  9. 2024LICO: Large Language Models for In-Context Molecular Optimizationcited by 26
  10. 2024FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearchcited by 23
  11. 2024Position: Leverage Foundational Models for Black-Box Optimizationcited by 21
  12. 2024Evolve Cost-aware Acquisition Functions Using Large Language Modelscited by 17
  13. 2024Reinforced In-Context Black-Box Optimizationcited by 14
  14. 2024Pretrained Optimization Model for Zero-Shot Black Box Optimizationcited by 13
  15. 2025Decoding-based Regressioncited by 12
  16. 2024Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Taskscited by 10
  17. 2025BOFormer: Learning to Solve Multi-Objective Bayesian Optimization via Non-Markovian RLcited by 10
  18. 2026AI Research Preference Modelscited by 10
  19. 2023Mnemosyne: Learning to Train Transformers with Transformerscited by 9
  20. 2023Transfer Learning for Bayesian Optimization on Heterogeneous Search Spacescited by 9
  21. 2024Language Model Embeddings Can Be Sufficient for Bayesian Optimizationcited by 8
  22. 2025PABBO: Preferential Amortized Black-Box Optimizationcited by 8
  23. 2025Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddingscited by 7
  24. 2024EARL-BO: Reinforcement Learning for Multi-Step Lookahead, High-Dimensional Bayesian Optimizationcited by 6
  25. 2025In-Context Multi-Objective Optimizationcited by 6
  26. 2026Agentic Bayesian Optimization through Surrogate-Augmented Autoresearchcited by 6
  27. 2024Principled Bayesian Optimisation in Collaboration with Human Expertscited by 5
  28. 2026AutoLLMResearch: Training Research Agents for Automating LLM Experiment Configuration - Learning from Cheap, Optimizing Expensivecited by 5
  29. 2022HyperBO+: Pre-training a universal prior for Bayesian optimization with hierarchical Gaussian processescited by 4
  30. 2024Principled Architecture-aware Scaling of Hyperparameterscited by 4
  31. 2025Put CASH on Bandits: A Max K-Armed Problem for Automated Machine Learningcited by 4
  32. 2024WEITS: A Wavelet-enhanced residual framework for interpretable time series forecastingcited by 3
  33. 2024Narrowing the Focus: Learned Optimizers for Pretrained Modelscited by 3
  34. 2025A Model Zoo of Vision Transformerscited by 3
  35. 2025None To Optima in Few Shots: Bayesian Optimization with MDP Priorscited by 3
  36. 2022Multi-step Planning for Automated Hyperparameter Optimization with OptFormercited by 2
  37. 2023MALIBO: Meta-learning for Likelihood-free Bayesian Optimizationcited by 2
  38. 2023Efficient Bayesian Optimization with Deep Kernel Learning and Transformer Pre-trained on Multiple Heterogeneous Datasetscited by 2

50 most cited of 81 All 81 (JSON)