# Consolidated learning -- a domain-specific model-free optimization strategy with examples for XGBoost and MIMIC-IV

arXiv: https://arxiv.org/abs/2201.11815 · HTML: https://mlsanity.com/paper/2201.11815/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 50 references that are in the map.
- First posted on arXiv 27 Jan 2022.

## Builds on (most cited first, 18 of 18)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously](https://mlsanity.com/paper/1801.01489/) (2018)
- [Non-stochastic Best Arm Identification and Hyperparameter Optimization](https://mlsanity.com/paper/1502.07943/) (2015)
- [Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning](https://mlsanity.com/paper/2007.04074/) (2020)
- [Towards a Neural Statistician](https://mlsanity.com/paper/1606.02185/) (2016)
- [Tunability: Importance of Hyperparameters of Machine Learning Algorithms](https://mlsanity.com/paper/1802.09596/) (2018)
- [BOCK : Bayesian Optimization with Cylindrical Kernels](https://mlsanity.com/paper/1806.01619/) (2018)
- [Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning](https://mlsanity.com/paper/1909.12552/) (2019)
- [Dataset2Vec: Learning Dataset Meta-Features](https://mlsanity.com/paper/1905.11063/) (2019)
- [LightAutoML: AutoML Solution for a Large Financial Services Ecosystem](https://mlsanity.com/paper/2109.01528/) (2021)
- [The Variational Homoencoder: Learning to learn high capacity generative models from few examples](https://mlsanity.com/paper/1807.08919/) (2018)
- [AutoPrognosis: Automated Clinical Prognostic Modeling via Bayesian Optimization with Structured Kernel Learning](https://mlsanity.com/paper/1802.07207/) (2018)
- [MIMIC-IF: Interpretability and Fairness Evaluation of Deep Learning Models on MIMIC-IV Dataset](https://mlsanity.com/paper/2102.06761/) (2021)
- [Machine learning for modeling the progression of Alzheimer disease dementia using clinical data: a systematic literature review](https://mlsanity.com/paper/2108.04174/) (2021)
- [Learning Multiple Defaults for Machine Learning Algorithms](https://mlsanity.com/paper/1811.09409/) (2018)
- [Towards explainable meta-learning](https://mlsanity.com/paper/2002.04276/) (2020)
- [MementoML: Performance of selected machine learning algorithm configurations on OpenML100 datasets](https://mlsanity.com/paper/2008.13162/) (2020)

## Built on by (most cited first, 2 of 2)

- [Transfer Learning with Deep Tabular Models](https://mlsanity.com/paper/2206.15306/) (2022)
- [SeFNet: Bridging Tabular Datasets with Semantic Feature Nets](https://mlsanity.com/paper/2306.11636/) (2023)

## Abstract (arXiv)

For many machine learning models, a choice of hyperparameters is a crucial step towards achieving high performance. Prevalent meta-learning approaches focus on obtaining good hyperparameters configurations with a limited computational budget for a completely new task based on the results obtained from the prior tasks. This paper proposes a new formulation of the tuning problem, called consolidated learning, more suited to practical challenges faced by model developers, in which a large number of predictive models are created on similar data sets. In such settings, we are interested in the total optimization time rather than tuning for a single task. We show that a carefully selected static portfolio of hyperparameters yields good results for anytime optimization, maintaining ease of use and implementation. Moreover, we point out how to construct such a portfolio for specific domains. The improvement in the optimization is possible due to more efficient transfer of hyperparameter configurations between similar tasks. We demonstrate the effectiveness of this approach through an empirical study for XGBoost algorithm and the collection of predictive tasks extracted from the MIMIC-IV medical database; however, consolidated learning is applicable in many others fields.