mlsanityData to 14 Oct 2026

Large Language Model Agent for Hyper-Parameter Optimization

arXiv 2402.01881 ↗First posted 2 Feb 2024cs.LG cs.AIcited by 64

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Hyperparameter optimization is critical in modern machine learning, requiring expert knowledge, numerous trials, and high computational and human resources. Despite the advancements in Automated Machine Learning (AutoML), challenges in terms of trial efficiency, setup complexity, and interoperability still persist. To address these issues, we introduce a novel paradigm leveraging Large Language Models (LLMs) to automate hyperparameter optimization across diverse machine learning tasks, which is named AgentHPO (short for LLM Agent-based Hyperparameter Optimization). Specifically, AgentHPO processes the task information autonomously, conducts experiments with specific hyperparameters (HPs), and iteratively optimizes them based on historical trials. This human-like optimization process largely reduces the number of required trials, simplifies the setup process, and enhances interpretability and user trust, compared to traditional AutoML methods. Extensive empirical experiments conducted on 12 representative machine-learning tasks indicate that AgentHPO not only matches but also often surpasses the best human trials in terms of performance while simultaneously providing explainable results. Further analysis sheds light on the strategies employed by the LLM in optimizing these tasks, highlighting its effectiveness and adaptability in various scenarios.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 30 of its 51 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

30 of 51 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework1 Aug

Other references · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  5. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  10. 2016Variational Graph Auto-Encoderscited by 1,382
  11. 2020LightGCN: Simplifying and Powering Graph Convolution Network for Recommendationcited by 1,139
  12. 2023Large Language Models as Optimizerscited by 864
17 more
  1. 2017DeepFM: A Factorization-Machine based Neural Network for CTR Predictioncited by 676
  2. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  3. 2016ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentationcited by 291
  4. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  5. 2023S3: Social-network Simulation System with Large Language Model-Empowered Agentscited by 233
  6. 2020On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practicecited by 182
  7. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 131
  8. 2023Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineeringcited by 127
  9. 2020Hyper-Parameter Optimization: A Review of Algorithms and Applicationscited by 114
  10. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 112
  11. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 104
  12. 2022Towards Learning Universal Hyperparameter Optimizers with Transformerscited by 81
  13. 2023AutoML-GPT: Automatic Machine Learning with GPTcited by 81
  14. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  15. 2023MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Taskscited by 66
  16. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 35
  17. 2023PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learningcited by 31

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 25; 2026: 31.

64 papers.

Sort:

By area:

  1. 2024Data Interpreter: An LLM Agent For Data Sciencecited by 225
  2. 2024AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoMLcited by 87
  3. 2024SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositoriescited by 52
  4. 2025Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedbackcited by 25
  5. 2024SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learningcited by 24
  6. 2024Sequential Large Language Model-Based Hyper-parameter Optimizationcited by 24
  7. 2025MLZero: A Multi-Agent System for End-to-end Machine Learning Automationcited by 24
  8. 2025Large Language Model-based Data Science Agent: A Surveycited by 23
  9. 2025Optuna vs Code Llama: Are LLMs a New Paradigm for Hyperparameter Tuning?cited by 21
  10. 2024Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Surveycited by 19
  11. 2024Large Language Model-driven Meta-structure Discovery in Heterogeneous Information Networkcited by 16
  12. 2025AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Taskscited by 16
38 more
  1. 2025ORFS-agent: Tool-Using Agents for Chip Design Optimizationcited by 14
  2. 2025Reinforcement Learning for Machine Learning Engineering Agentscited by 14
  3. 2025From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systemscited by 8
  4. 2025AD-AGENT: A Multi-agent Framework for End-to-end Anomaly Detectioncited by 8
  5. 2026TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Explorationcited by 8
  6. 2026Agentic Bayesian Optimization through Surrogate-Augmented Autoresearchcited by 6
  7. 2026AgenticRecTune: Multi-Agent with Self-Evolving Skillhub for Recommendation System Optimizationcited by 5
  8. 2026AutoLLMResearch: Training Research Agents for Automating LLM Experiment Configuration - Learning from Cheap, Optimizing Expensivecited by 5
  9. 2026When Is an LLM Worth It for Hyperparameter Optimization? A Budget-Matched Study on Tabular Data Finds the Warm-Start Is a Default Configuration, Not the Modelcited by 5
  10. 2024SemiKong: Curating, Training, and Evaluating A Semiconductor Industry-Specific Large Language Modelcited by 4
  11. 2025Generalizing Test-time Compute-optimal Scaling as an Optimizable Graphcited by 4
  12. 2025OptiMindTune: A Multi-Agent Framework for Intelligent Hyperparameter Optimizationcited by 3
  13. 2025LLMs as In-Context Meta-Learners for Model and Hyperparameter Selectioncited by 3
  14. 2026Why Solve It Twice? Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineeringcited by 3
  15. 2025MoColl: Agent-Based Specific and General Model Collaboration for Image Captioningcited by 2
  16. 2025MetaLLMix : An XAI Aided LLM-Meta-learning Based Approach for Hyper-parameters Optimizationcited by 2
  17. 2025Small LLMs with Expert Blocks Are Good Enough for Hyperparamter Tuningcited by 2
  18. 2026A Language-Guided Bayesian Optimization for Efficient LoRA Hyperparameter Searchcited by 2
  19. 2026Matryoshka Agent: Unfolding Sub-Agents for Long-Horizon Machine Learning Engineeringcited by 2
  20. 2025REMoH: A Reflective Evolution of Multi-objective Heuristics approach via Large Language Modelscited by 1
  21. 2025DaMoC: Efficiently Selecting the Optimal Large Language Model for Fine-tuning Domain Tasks Based on Data and Model Compressioncited by 1
  22. 2026From Bits to Chips: An LLM-based Hardware-Aware Quantization Agent for Streamlined Deployment of LLMscited by 1
  23. 2026ARMOR: Agentic Reasoning for Methods Orchestration and Reparameterization for Robust Adversarial Attackscited by 1
  24. 2026Rudder: Steering Prefetching in Distributed GNN Training using LLM Agentscited by 1
  25. 2026EPOCH: An Agentic Protocol for Multi-Round System Optimizationcited by 1
  26. 2026Remember, Don't Re-read: Stateful ReAct Agents for Token-Efficient Autonomous Experimentationcited by 1
  27. 2026AI Training Manager: Bounded Closed-Loop Control of Adaptive Training Recipescited by 1
  28. 2025Large Language Model Enhanced Particle Swarm Optimization for Hyperparameter Tuning for Deep Learning Modelscited by 0
  29. 2025Optimization Problem Solving Can Transition to Evolutionary Agentic Workflowscited by 0
  30. 2025StructSynth: Dependency Graphs as Generation Plans for Low-Data Tabular Synthesis with Language Modelscited by 0
  31. 2025Language-Guided Tuning: Configuration Optimization for Automated ML Researchcited by 0
  32. 2025Systematic Optimization of Open Source Large Language Models for Mathematical Reasoningcited by 0
  33. 2025Agent2: An Agent-Generates-Agent Framework for Reinforcement Learning Automationcited by 0
  34. 2025Pre-Hoc Predictions in AutoML: Leveraging LLMs to Enhance Model Selection and Benchmarking for Tabular datasetscited by 0
  35. 2025Auxiliary-Hyperparameter-Free Sampling: Entropy Equilibrium for Text Generationcited by 0
  36. 2026A Meta-Knowledge-Augmented LLM Framework for Hyperparameter Optimization in Time-Series Forecastingcited by 0
  37. 2026CoFEH: LLM-driven Feature Engineering Empowered by Collaborative Bayesian Hyperparameter Optimizationcited by 0
  38. 2026AutoAdapt: An Automated Domain Adaptation Framework for LLMscited by 0

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch
  2. 2026AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers
  3. 2025OPT-BENCH: Evaluating LLM Agent on Large-Scale Search Spaces Optimization Problems
  4. 2025AutoPDL: Automatic Prompt Optimization for LLM Agents
  5. 2026LLM Agents for Automated Post-Training via Structured Recipe Optimization
  6. 2025AutoEP: LLMs-Driven Automation of Hyperparameter Evolution for Metaheuristic Algorithms
  7. 2026Efficient Hyperparameter Optimization for LLM Reinforcement Learning
  8. 2024Offline Training of Language Model Agents with Functions as Learnable Weights