# Assembled-OpenML: Creating Efficient Benchmarks for Ensembles in AutoML with OpenML

arXiv: https://arxiv.org/abs/2307.00285 · HTML: https://mlsanity.com/paper/2307.00285/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 73 references that are in the map.
- First posted on arXiv 1 Jul 2023.

## Builds on (most cited first, 13 of 13)

- [OpenML: networked science in machine learning](https://mlsanity.com/paper/1407.7722/) (2014)
- [AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data](https://mlsanity.com/paper/2003.06505/) (2020)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [Caffe: Convolutional Architecture for Fast Feature Embedding](https://mlsanity.com/paper/1408.5093/) (2014)
- [OpenML Benchmarking Suites](https://mlsanity.com/paper/1708.03731/) (2017)
- [Automated Algorithm Selection: Survey and Perspectives](https://mlsanity.com/paper/1811.11597/) (2018)
- [OpenML-Python: an extensible Python API for OpenML](https://mlsanity.com/paper/1911.02490/) (2019)
- [Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization](https://mlsanity.com/paper/1905.04970/) (2019)
- [Autostacker: A Compositional Evolutionary Learning System](https://mlsanity.com/paper/1803.00684/) (2018)
- [Automatic Exploration of Machine Learning Experiments on OpenML](https://mlsanity.com/paper/1806.10961/) (2018)
- [Extreme Algorithm Selection With Dyadic Feature Representation](https://mlsanity.com/paper/2001.10741/) (2020)
- [AutoDES: AutoML Pipeline Generation of Classification with Dynamic Ensemble Strategy Selection](https://mlsanity.com/paper/2201.00207/) (2022)
- [Transformative Machine Learning](https://mlsanity.com/paper/1811.03392/) (2018)

## Built on by (most cited first, 9 of 9)

- [TabRepo: A Large Scale Repository of Tabular Model Evaluations and its AutoML Applications](https://mlsanity.com/paper/2311.02971/) (2023)
- [Q(D)O-ES: Population-based Quality (Diversity) Optimisation for Post Hoc Ensemble Selection in AutoML](https://mlsanity.com/paper/2307.08364/) (2023)
- [PSEO: Optimizing Post-hoc Stacking Ensemble Through Hyperparameter Tuning](https://mlsanity.com/paper/2508.05144/) (2025)
- [Ensembling Finetuned Language Models for Text Classification](https://mlsanity.com/paper/2410.19889/) (2024)
- [TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks](https://mlsanity.com/paper/2606.02384/) (2026)
- [Regularized Neural Ensemblers](https://mlsanity.com/paper/2410.04520/) (2024)
- [HAPEns: Hardware-Aware Post-Hoc Ensembling for Tabular Data](https://mlsanity.com/paper/2603.10582/) (2026)
- [Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters](https://mlsanity.com/paper/2607.05104/) (2026)
- [Ensembles at Any Cost? Accuracy-Energy Trade-offs in Recommender Systems](https://mlsanity.com/paper/2604.07869/) (2026)

## Abstract (arXiv)

Automated Machine Learning (AutoML) frameworks regularly use ensembles. Developers need to compare different ensemble techniques to select appropriate techniques for an AutoML framework from the many potential techniques. So far, the comparison of ensemble techniques is often computationally expensive, because many base models must be trained and evaluated one or multiple times. Therefore, we present Assembled-OpenML. Assembled-OpenML is a Python tool, which builds meta-datasets for ensembles using OpenML. A meta-dataset, called Metatask, consists of the data of an OpenML task, the task's dataset, and prediction data from model evaluations for the task. We can make the comparison of ensemble techniques computationally cheaper by using the predictions stored in a metatask instead of training and evaluating base models. To introduce Assembled-OpenML, we describe the first version of our tool. Moreover, we present an example of using Assembled-OpenML to compare a set of ensemble techniques. For this example comparison, we built a benchmark using Assembled-OpenML and implemented ensemble techniques expecting predictions instead of base models as input. In our example comparison, we gathered the prediction data of 1523 base models for 31 datasets. Obtaining the prediction data for all base models using Assembled-OpenML took ~ 1 hour in total. In comparison, obtaining the prediction data by training and evaluating just one base model on the most computationally expensive dataset took ~ 37 minutes.