# Privileged Zero-Shot AutoML

arXiv: https://arxiv.org/abs/2106.13743 · HTML: https://mlsanity.com/paper/2106.13743/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 25 Jun 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [Probabilistic Matrix Factorization for Automated Machine Learning](https://mlsanity.com/paper/1705.05355/) (2017)
- [OBOE: Collaborative Filtering for AutoML Model Selection](https://mlsanity.com/paper/1808.03233/) (2018)
- [Autostacker: A Compositional Evolutionary Learning System](https://mlsanity.com/paper/1803.00684/) (2018)
- [AutoML using Metadata Language Embeddings](https://mlsanity.com/paper/1910.03698/) (2019)
- [Automatic Machine Learning by Pipeline Synthesis using Model-Based Reinforcement Learning and a Grammar](https://mlsanity.com/paper/1905.10345/) (2019)
- [Fast Task-Aware Architecture Inference](https://mlsanity.com/paper/1902.05781/) (2019)

## Built on by (most cited first, 4 of 4)

- [Towards Learning Universal Hyperparameter Optimizers with Transformers](https://mlsanity.com/paper/2205.13320/) (2022)
- [Towards Green Automated Machine Learning: Status Quo and Future Directions](https://mlsanity.com/paper/2111.05850/) (2021)
- [Mining Robust Default Configurations for Resource-constrained AutoML](https://mlsanity.com/paper/2202.09927/) (2022)
- [Introducing "Forecast Utterance" for Conversational Data Science](https://mlsanity.com/paper/2309.03877/) (2023)

## Abstract (arXiv)

This work improves the quality of automated machine learning (AutoML) systems by using dataset and function descriptions while significantly decreasing computation time from minutes to milliseconds by using a zero-shot approach. Given a new dataset and a well-defined machine learning task, humans begin by reading a description of the dataset and documentation for the algorithms to be used. This work is the first to use these textual descriptions, which we call privileged information, for AutoML. We use a pre-trained Transformer model to process the privileged text and demonstrate that using this information improves AutoML performance. Thus, our approach leverages the progress of unsupervised representation learning in natural language processing to provide a significant boost to AutoML. We demonstrate that using only textual descriptions of the data and functions achieves reasonable classification performance, and adding textual descriptions to data meta-features improves classification across tabular datasets. To achieve zero-shot AutoML we train a graph neural network with these description embeddings and the data meta-features. Each node represents a training dataset, which we use to predict the best machine learning pipeline for a new test dataset in a zero-shot fashion. Our zero-shot approach rapidly predicts a high-quality pipeline for a supervised learning task and dataset. In contrast, most AutoML systems require tens or hundreds of pipeline evaluations. We show that zero-shot AutoML reduces running and prediction times from minutes to milliseconds, consistently across datasets. By speeding up AutoML by orders of magnitude this work demonstrates real-time AutoML.