mlsanityData to 7 Oct 2026

Using Large Language Models for Hyperparameter Optimization

arXiv 2312.04528 ↗First posted 7 Dec 2023cs.LG cs.AIcited by 71

Not on the map. Why?

Abstract (arXiv)

This paper explores the use of foundational large language models (LLMs) in hyperparameter optimization (HPO). Hyperparameters are critical in determining the effectiveness of machine learning models, yet their optimization often relies on manual approaches in limited-budget settings. By prompting LLMs with dataset and model descriptions, we develop a methodology where LLMs suggest hyperparameter configurations, which are iteratively refined based on model performance. Our empirical evaluations on standard benchmarks reveal that within constrained search budgets, LLMs can match or outperform traditional HPO methods like Bayesian optimization across different models on standard benchmarks. Furthermore, we propose to treat the code specifying our model as a hyperparameter, which the LLM outputs and affords greater flexibility than existing HPO approaches.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Builds on 45 of its 81 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

45 of 81 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  11. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  12. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
33 more
  1. 2023Large Language Models as Optimizerscited by 830
  2. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  3. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  4. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  5. 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 357
  6. 2017Population Based Training of Neural Networkscited by 351
  7. 2019Optimizing Millions of Hyperparameters by Implicit Differentiationcited by 303
  8. 2017Forward and Reverse Gradient-Based Hyperparameter Optimizationcited by 232
  9. 2021SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimizationcited by 229
  10. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 219
  11. 2020On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practicecited by 181
  12. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  13. 2021Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challengescited by 143
  14. 2016Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasetscited by 131
  15. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  16. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 128
  17. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 111
  18. 2019Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functionscited by 104
  19. 2021Multi-Objective Bayesian Optimization over High-Dimensional Search Spacescited by 87
  20. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  21. 2023AutoML-GPT: Automatic Machine Learning with GPTcited by 80
  22. 2023Can GPT-4 Perform Neural Architecture Search?cited by 74
  23. 2023Eight Things to Know about Large Language Modelscited by 73
  24. 2018Stochastic Hyperparameter Optimization through Hypernetworkscited by 72
  25. 2019Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonflycited by 66
  26. 2021Prioritized Architecture Sampling with Monto-Carlo Tree Searchcited by 32
  27. 2022On Implicit Bias in Overparameterized Bilevel Optimizationcited by 30
  28. 2021Meta-Learning to Improve Pre-Trainingcited by 27
  29. 2022Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curvecited by 18
  30. 2020Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobianscited by 17
  31. 2019Towards Assessing the Impact of Bayesian Optimization's Own Hyperparameterscited by 16
  32. 2023Unlocking the Potential of ChatGPT: A Comprehensive Exploration of its Applications, Advantages, Limitations, and Future Directions in Natural Language Processingcited by 11
  33. 2022Task Selection for AutoML System Evaluationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 23; 2026: 22.

71 papers.

Sort:

By area:

  1. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 78
  2. 2024A Systematic Survey on Large Language Models for Algorithm Designcited by 65
  3. 2024Large Language Model Agent for Hyper-Parameter Optimizationcited by 62
  4. 2024Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMscited by 62
  5. 2024Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoningcited by 48
  6. 2024Offline Training of Language Model Agents with Functions as Learnable Weightscited by 47
  7. 2024LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generationcited by 45
  8. 2024Large Language Model-Based Evolutionary Optimizer: Reasoning with elitismcited by 27
  9. 2024LICO: Large Language Models for In-Context Molecular Optimizationcited by 26
  10. 2024Sequential Large Language Model-Based Hyper-parameter Optimizationcited by 23
  11. 2024REvolve: Reward Evolution with Large Language Models using Human Feedbackcited by 22
  12. 2024FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearchcited by 22
38 more
  1. 2025MLZero: A Multi-Agent System for End-to-end Machine Learning Automationcited by 22
  2. 2024Position: Leverage Foundational Models for Black-Box Optimizationcited by 21
  3. 2025Optuna vs Code Llama: Are LLMs a New Paradigm for Hyperparameter Tuning?cited by 21
  4. 2024When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challengescited by 20
  5. 2024Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Surveycited by 19
  6. 2026Towards Execution-Grounded Automated AI Researchcited by 19
  7. 2024Evolve Cost-aware Acquisition Functions Using Large Language Modelscited by 17
  8. 2026Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearchcited by 12
  9. 2024Scattered Forest Search: Smarter Code Space Exploration with LLMscited by 10
  10. 2024Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Taskscited by 8
  11. 2024LLMs for Cold-Start Cutting Plane Separator Configurationcited by 8
  12. 2024NADER: Neural Architecture Design via Multi-Agent Collaborationcited by 8
  13. 2025DatawiseAgent: A Notebook-Centric LLM Agent Framework for Adaptive and Robust Data Science Automationcited by 8
  14. 2024Large Language Models as In-context AI Generators for Quality-Diversitycited by 7
  15. 2024JacNet: Learning Functions with Structured Jacobianscited by 7
  16. 2025Improving LLM-based Global Optimization with Search Space Partitioningcited by 6
  17. 2024Improving Hyperparameter Optimization with Checkpointed Model Weightscited by 5
  18. 2025Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyondcited by 5
  19. 2025Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Modelscited by 5
  20. 2026Agentic Bayesian Optimization through Surrogate-Augmented Autoresearchcited by 5
  21. 2025Autonomous Radiotherapy Treatment Planning Using DOLA: A Privacy-Preserving, LLM-Based Optimization Agentcited by 4
  22. 2025Large Language Models in Operations Research: Methods, Applications, and Challengescited by 4
  23. 2026AutoLLMResearch: Training Research Agents for Automating LLM Experiment Configuration - Learning from Cheap, Optimizing Expensivecited by 4
  24. 2024Language Models as Efficient Reward Function Searchers for Custom-Environment Multi-Objective Reinforcementcited by 3
  25. 2025LLM4GNAS: A Large Language Model Based Toolkit for Graph Neural Architecture Searchcited by 3
  26. 2025Generative Reliability-Based Design Optimization Using In-Context Learning Capabilities of Large Language Modelscited by 3
  27. 2025LLMs as In-Context Meta-Learners for Model and Hyperparameter Selectioncited by 3
  28. 2024Large Language Models for Tuning Evolution Strategiescited by 2
  29. 2024Adaptive Augmentation Policy Optimization with LLM Feedbackcited by 2
  30. 2025MoColl: Agent-Based Specific and General Model Collaboration for Image Captioningcited by 2
  31. 2025DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Rewardcited by 2
  32. 2025MetaLLMix : An XAI Aided LLM-Meta-learning Based Approach for Hyper-parameters Optimizationcited by 2
  33. 2025Small LLMs with Expert Blocks Are Good Enough for Hyperparamter Tuningcited by 2
  34. 2025GRPOformer: Advancing Hyperparameter Optimization via Group Relative Policy Optimizationcited by 2
  35. 2026A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Searchcited by 2
  36. 2026AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizerscited by 2
  37. 2025REMoH: A Reflective Evolution of Multi-objective Heuristics approach via Large Language Modelscited by 1
  38. 2026From Bits to Chips: An LLM-based Hardware-Aware Quantization Agent for Streamlined Deployment of LLMscited by 1

50 most cited of 71 All 71 (JSON)