mlsanityData to 14 Oct 2026

LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection

arXiv 2510.26510 ↗First posted 30 Oct 2025cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

Model and hyperparameter selection are critical but challenging in machine learning, typically requiring expert intuition or expensive automated search. We investigate whether large language models (LLMs) can act as in-context meta-learners for this task. By converting each dataset into interpretable metadata, we prompt an LLM to recommend both model families and hyperparameters. We study two prompting strategies: (1) a zero-shot mode relying solely on pretrained knowledge, and (2) a meta-informed mode augmented with examples of models and their performance on past tasks. Across synthetic and real-world benchmarks, we show that LLMs can exploit dataset metadata to recommend competitive models and hyperparameters without search, and that improvements from meta-informed prompting demonstrate their capacity for in-context meta-learning. These results highlight a promising new role for LLMs as lightweight, general-purpose assistants for model selection and hyperparameter optimization.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 22 of its 42 references that are in the map. First posted on arXiv 30 Oct 2025.

Builds on

22 of 42 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  4. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 699
  5. 2015Non-stochastic Best Arm Identification and Hyperparameter Optimizationcited by 221
  6. 2023Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineeringcited by 127
  7. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
  8. 2020HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisationcited by 83
  9. 2020Meta-Learning with Adaptive Hyperparameterscited by 79
  10. 2023Can GPT-4 Perform Neural Architecture Search?cited by 74
  11. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  12. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
10 more
  1. 2024Large Language Model Agent for Hyper-Parameter Optimizationcited by 64
  2. 2023Meta-in-context learning in large language modelscited by 59
  3. 2020Efficient Automatic CASH via Rising Banditscited by 31
  4. 2023PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learningcited by 31
  5. 2023Primacy Effect of ChatGPTcited by 27
  6. 2024Sequential Large Language Model-Based Hyper-parameter Optimizationcited by 24
  7. 2020Large Dimensional Analysis and Improvement of Multi Task Learningcited by 9
  8. 2025Put CASH on Bandits: A Max K-Armed Problem for Automated Machine Learningcited by 4
  9. 2023A systematic study comparing hyperparameter optimization engines on tabular datacited by 2
  10. 2023A Meta-Level Learning Algorithm for Sequential Hyper-Parameter Space Reduction in AutoMLcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026A Meta-Knowledge-Augmented LLM Framework for Hyperparameter Optimization in Time-Series Forecastingcited by 0
  2. 2026Meta-Learning Preferences for Multilingual LLM Alignmentcited by 0
  3. 2026DLC: A Metric-Guided Dynamic Loss Controller for Multi-Objective Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Prompt-MII: Meta-Learning Instruction Induction for LLMs
  2. 2026Is One-Shot In-Context Learning Helpful for Data Selection in Task-Specific Fine-Tuning of Multimodal LLMs?
  3. 2024The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting
  4. 2023In-Context Explainers: Harnessing LLMs for Explaining Black Box Models
  5. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
  6. 2026MetaToolAgent: Towards Generalizable Tool Usage in LLMs through Meta-Learning
  7. 2024Efficient LLM Context Distillation
  8. 2024How to Train Data-Efficient LLMs