# LLMs as In-Context Meta-Learners for Model and Hyperparameter Selection

arXiv: https://arxiv.org/abs/2510.26510 · HTML: https://mlsanity.com/paper/2510.26510/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 30 Oct 2025.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization](https://mlsanity.com/paper/1603.06560/) (2016)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering](https://mlsanity.com/paper/2305.03403/) (2023)
- [General-Purpose In-Context Learning by Meta-Learning Transformers](https://mlsanity.com/paper/2212.04458/) (2022)
- [HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisation](https://mlsanity.com/paper/2012.03826/) (2020)
- [Meta-Learning with Adaptive Hyperparameters](https://mlsanity.com/paper/2011.00209/) (2020)
- [Can GPT-4 Perform Neural Architecture Search?](https://mlsanity.com/paper/2304.10970/) (2023)
- [Using Large Language Models for Hyperparameter Optimization](https://mlsanity.com/paper/2312.04528/) (2023)
- [Eliminating Position Bias of Language Models: A Mechanistic Approach](https://mlsanity.com/paper/2407.01100/) (2024)
- [Large Language Model Agent for Hyper-Parameter Optimization](https://mlsanity.com/paper/2402.01881/) (2024)
- [Meta-in-context learning in large language models](https://mlsanity.com/paper/2305.12907/) (2023)
- [Efficient Automatic CASH via Rising Bandits](https://mlsanity.com/paper/2012.04371/) (2020)
- [PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning](https://mlsanity.com/paper/2306.12370/) (2023)
- [Primacy Effect of ChatGPT](https://mlsanity.com/paper/2310.13206/) (2023)
- [Sequential Large Language Model-Based Hyper-parameter Optimization](https://mlsanity.com/paper/2410.20302/) (2024)
- [Large Dimensional Analysis and Improvement of Multi Task Learning](https://mlsanity.com/paper/2009.01591/) (2020)
- [Put CASH on Bandits: A Max K-Armed Problem for Automated Machine Learning](https://mlsanity.com/paper/2505.05226/) (2025)
- [A systematic study comparing hyperparameter optimization engines on tabular data](https://mlsanity.com/paper/2311.15854/) (2023)
- [A Meta-Level Learning Algorithm for Sequential Hyper-Parameter Space Reduction in AutoML](https://mlsanity.com/paper/2312.06305/) (2023)

## Built on by (most cited first, 3 of 3)

- [A Meta-Knowledge-Augmented LLM Framework for Hyperparameter Optimization in Time-Series Forecasting](https://mlsanity.com/paper/2602.01445/) (2026)
- [Meta-Learning Preferences for Multilingual LLM Alignment](https://mlsanity.com/paper/2607.13315/) (2026)
- [DLC: A Metric-Guided Dynamic Loss Controller for Multi-Objective Training](https://mlsanity.com/paper/2610.11433/) (2026)

## Abstract (arXiv)

Model and hyperparameter selection are critical but challenging in machine learning, typically requiring expert intuition or expensive automated search. We investigate whether large language models (LLMs) can act as in-context meta-learners for this task. By converting each dataset into interpretable metadata, we prompt an LLM to recommend both model families and hyperparameters. We study two prompting strategies: (1) a zero-shot mode relying solely on pretrained knowledge, and (2) a meta-informed mode augmented with examples of models and their performance on past tasks. Across synthetic and real-world benchmarks, we show that LLMs can exploit dataset metadata to recommend competitive models and hyperparameters without search, and that improvements from meta-informed prompting demonstrate their capacity for in-context meta-learning. These results highlight a promising new role for LLMs as lightweight, general-purpose assistants for model selection and hyperparameter optimization.