# Meta-Learning for Symbolic Hyperparameter Defaults

arXiv: https://arxiv.org/abs/2106.05767 · HTML: https://mlsanity.com/paper/2106.05767/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 57 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 16 of 16)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R](https://mlsanity.com/paper/1508.04409/) (2015)
- [Tunability: Importance of Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/1802.09596/) (2018)
- [An Open Source AutoML Benchmark](https://mlsanity.com/paper/1907.00909/) (2019)
- [Hyperparameter Importance Across Datasets](https://mlsanity.com/paper/1710.04725/) (2017)
- [To tune or not to tune the number of trees in random forest?](https://mlsanity.com/paper/1705.05654/) (2017)
- [Using a thousand optimization tasks to learn hyperparameter search strategies](https://mlsanity.com/paper/2002.11887/) (2020)
- [Importance of Tuning Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/2007.07588/) (2020)
- [Warmstarting of Model-based Algorithm Configuration](https://mlsanity.com/paper/1709.04636/) (2017)
- [Automatic Exploration of Machine Learning Experiments on OpenML](https://mlsanity.com/paper/1806.10961/) (2018)
- [Learning Multiple Defaults for Machine Learning Algorithms](https://mlsanity.com/paper/1811.09409/) (2018)
- [Practical and sample efficient zero-shot HPO](https://mlsanity.com/paper/2007.13382/) (2020)
- [Evolving Multimodal Robot Behavior via Many Stepping Stones with the Combinatorial Multi-Objective Evolutionary Algorithm](https://mlsanity.com/paper/1807.03392/) (2018)
- [Rethinking Default Values: a Low Cost and Efficient Strategy to Define Hyperparameters](https://mlsanity.com/paper/2008.00025/) (2020)

## Built on by (most cited first, 4 of 4)

- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimization](https://mlsanity.com/paper/2109.03670/) (2021)
- [Mining Robust Default Configurations for Resource-constrained AutoML](https://mlsanity.com/paper/2202.09927/) (2022)
- [Heterogeneity-Oblivious Robust Federated Learning](https://mlsanity.com/paper/2508.03579/) (2025)

## Abstract (arXiv)

Hyperparameter optimization in machine learning (ML) deals with the problem of empirically learning an optimal algorithm configuration from data, usually formulated as a black-box optimization problem. In this work, we propose a zero-shot method to meta-learn symbolic default hyperparameter configurations that are expressed in terms of the properties of the dataset. This enables a much faster, but still data-dependent, configuration of the ML algorithm, compared to standard hyperparameter optimization approaches. In the past, symbolic and static default values have usually been obtained as hand-crafted heuristics. We propose an approach of learning such symbolic configurations as formulas of dataset properties from a large set of prior evaluations on multiple datasets by optimizing over a grammar of expressions using an evolutionary algorithm. We evaluate our method on surrogate empirical performance models as well as on real data across 6 ML algorithms on more than 100 datasets and demonstrate that our method indeed finds viable symbolic defaults.