mlsanityData to 14 Oct 2026

Small LLMs with Expert Blocks Are Good Enough for Hyperparamter Tuning

arXiv 2509.15561 ↗First posted 19 Sep 2025cs.LG cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Hyper-parameter Tuning (HPT) is a necessary step in machine learning (ML) pipelines but becomes computationally expensive and opaque with larger models. Recently, Large Language Models (LLMs) have been explored for HPT, yet most rely on models exceeding 100 billion parameters. We propose an Expert Block Framework for HPT using Small LLMs. At its core is the Trajectory Context Summarizer (TCS), a deterministic block that transforms raw training trajectories into structured context, enabling small LLMs to analyze optimization progress with reliability comparable to larger models. Using two locally-run LLMs (phi4:reasoning14B and qwen2.5-coder:32B) and a 10-trial budget, our TCS-enabled HPT pipeline achieves average performance within ~0.9 percentage points of GPT-4 across six diverse tasks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 10 of its 20 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

10 of 20 references.

References · 10

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  3. 2023Tree-Structured Parzen Estimator: Understanding Its Algorithm Components and Their Roles for Better Empirical Performancecited by 147
  4. 2021HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPOcited by 112
  5. 2023AutoML-GPT: Automatic Machine Learning with GPTcited by 81
  6. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  7. 2023MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Taskscited by 66
  8. 2024Large Language Model Agent for Hyper-Parameter Optimizationcited by 64
  9. 2024Sequential Large Language Model-Based Hyper-parameter Optimizationcited by 24
  10. 2025Optuna vs Code Llama: Are LLMs a New Paradigm for Hyperparameter Tuning?cited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026When Is an LLM Worth It for Hyperparameter Optimization? A Budget-Matched Study on Tabular Data Finds the Warm-Start Is a Default Configuration, Not the Modelcited by 5
  2. 2026AI Training Manager: Bounded Closed-Loop Control of Adaptive Training Recipescited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Can LLMs Beat Classical Hyperparameter Optimization Algorithms? A Study on autoresearch
  2. 2025Predictable Scale: Part I, Step Law -- Optimal Hyperparameter Scaling Law in Large Language Model Pretraining
  3. 2025LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection
  4. 2025The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements
  5. 2025Rank Also Matters: Hierarchical Configuration for Mixture of Adapter Experts in LLM Fine-Tuning
  6. 2025Beyond Single-Task: Robust Multi-Task Length Generalization for LLMs
  7. 2025ExpertSteer: Intervening in LLMs through Expert Knowledge
  8. 2024Low-Rank Quantization-Aware Training for LLMs