# On Taking Advantage of Opportunistic Meta-knowledge to Reduce Configuration Spaces for Automated Machine Learning

arXiv: https://arxiv.org/abs/2208.04376 · HTML: https://mlsanity.com/paper/2208.04376/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 39 references that are in the map.
- First posted on arXiv 8 Aug 2022.

## Builds on (most cited first, 15 of 15)

- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [Tunability: Importance of Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/1802.09596/) (2018)
- [How Complex is your classification problem? A survey on measuring classification complexity](https://mlsanity.com/paper/1808.03591/) (2018)
- [Hyperparameter Importance Across Datasets](https://mlsanity.com/paper/1710.04725/) (2017)
- [Importance of Tuning Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/2007.07588/) (2020)
- [Benchmark and Survey of Automated Machine Learning Frameworks](https://mlsanity.com/paper/1904.12054/) (2019)
- [Automated Deep Learning: Neural Architecture Search Is Not the End](https://mlsanity.com/paper/2112.09245/) (2021)
- [AutonoML: Towards an Integrated Framework for Autonomous Machine Learning](https://mlsanity.com/paper/2012.12600/) (2020)
- [Automatic Composition and Optimization of Multicomponent Predictive Systems With an Extended Auto-WEKA](https://mlsanity.com/paper/1612.08789/) (2016)
- [Layered TPOT: Speeding up Tree-based Pipeline Optimization](https://mlsanity.com/paper/1801.06007/) (2018)
- [A Review of Meta-level Learning in the Context of Multi-component, Multi-level Evolving Prediction Systems](https://mlsanity.com/paper/2007.10818/) (2020)
- [AVATAR -- Machine Learning Pipeline Evaluation Using Surrogate Model](https://mlsanity.com/paper/2001.11158/) (2020)
- [AutoWeka4MCPS-AVATAR: Accelerating Automated Machine Learning Pipeline Composition and Optimisation](https://mlsanity.com/paper/2011.11846/) (2020)
- [The Roles and Modes of Human Interactions with Automated Machine Learning Systems](https://mlsanity.com/paper/2205.04139/) (2022)
- [Exploring Opportunistic Meta-knowledge to Reduce Search Spaces for Automated Machine Learning](https://mlsanity.com/paper/2105.00282/) (2021)

## Abstract (arXiv)

The automated machine learning (AutoML) process can require searching through complex configuration spaces of not only machine learning (ML) components and their hyperparameters but also ways of composing them together, i.e. forming ML pipelines. Optimisation efficiency and the model accuracy attainable for a fixed time budget suffer if this pipeline configuration space is excessively large. A key research question is whether it is both possible and practical to preemptively avoid costly evaluations of poorly performing ML pipelines by leveraging their historical performance for various ML tasks, i.e. meta-knowledge. The previous experience comes in the form of classifier/regressor accuracy rankings derived from either (1) a substantial but non-exhaustive number of pipeline evaluations made during historical AutoML runs, i.e. 'opportunistic' meta-knowledge, or (2) comprehensive cross-validated evaluations of classifiers/regressors with default hyperparameters, i.e. 'systematic' meta-knowledge. Numerous experiments with the AutoWeka4MCPS package suggest that (1) opportunistic/systematic meta-knowledge can improve ML outcomes, typically in line with how relevant that meta-knowledge is, and (2) configuration-space culling is optimal when it is neither too conservative nor too radical. However, the utility and impact of meta-knowledge depend critically on numerous facets of its generation and exploitation, warranting extensive analysis; these are often overlooked/underappreciated within AutoML and meta-learning literature. In particular, we observe strong sensitivity to the `challenge' of a dataset, i.e. whether specificity in choosing a predictor leads to significantly better performance. Ultimately, identifying `difficult' datasets, thus defined, is crucial to both generating informative meta-knowledge bases and understanding optimal search-space reduction strategies.