# Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019

arXiv: https://arxiv.org/abs/2201.03801 · HTML: https://mlsanity.com/paper/2201.03801/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 81 references that are in the map.
- First posted on arXiv 11 Jan 2022.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Large-Scale Evolution of Image Classifiers](https://mlsanity.com/paper/1703.01041/) (2017)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)
- [Meta-Learning: A Survey](https://mlsanity.com/paper/1810.03548/) (2018)
- [Fast AutoAugment](https://mlsanity.com/paper/1905.00397/) (2019)
- [AutoML-Zero: Evolving Machine Learning Algorithms From Scratch](https://mlsanity.com/paper/2003.03384/) (2020)
- [NAS evaluation is frustratingly hard](https://mlsanity.com/paper/1912.12522/) (2019)
- [AdaNet: Adaptive Structural Learning of Artificial Neural Networks](https://mlsanity.com/paper/1607.01097/) (2016)
- [Utterance-level Aggregation For Speaker Recognition In The Wild](https://mlsanity.com/paper/1902.10107/) (2019)
- [Probabilistic Matrix Factorization for Automated Machine Learning](https://mlsanity.com/paper/1705.05355/) (2017)
- [DeepArchitect: Automatically Designing and Training Deep Architectures](https://mlsanity.com/paper/1704.08792/) (2017)

## Built on by (most cited first, 16 of 16)

- [Data Interpreter: An LLM Agent For Data Science](https://mlsanity.com/paper/2402.18679/) (2024)
- [NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies](https://mlsanity.com/paper/2210.03230/) (2022)
- [Meta-Album: Multi-domain Meta-Dataset for Few-Shot Image Classification](https://mlsanity.com/paper/2302.08909/) (2023)
- [Zero-Shot AutoML with Pretrained Models](https://mlsanity.com/paper/2206.08476/) (2022)
- [Lessons learned from the NeurIPS 2021 MetaDL challenge: Backbone fine-tuning without episodic meta-learning dominates for few-shot learning image classification](https://mlsanity.com/paper/2206.08138/) (2022)
- [Task Selection for AutoML System Evaluation](https://mlsanity.com/paper/2208.12754/) (2022)
- [Asynchronous Decentralized Bayesian Optimization for Large Scale Hyperparameter Optimization](https://mlsanity.com/paper/2207.00479/) (2022)
- [Advances in MetaDL: AAAI 2021 challenge and workshop](https://mlsanity.com/paper/2202.01890/) (2022)
- [NeurIPS'22 Cross-Domain MetaDL competition: Design and baseline results](https://mlsanity.com/paper/2208.14686/) (2022)
- [FAIR Universe HiggsML Uncertainty Dataset and Competition](https://mlsanity.com/paper/2410.02867/) (2024)
- [FAIR Universe Weak Lensing ML Uncertainty Challenge: Handling Uncertainties and Distribution Shifts for Precision Cosmology](https://mlsanity.com/paper/2604.14451/) (2026)
- [Meta-learning from Learning Curves Challenge: Lessons learned from the First Round and Design of the Second Round](https://mlsanity.com/paper/2208.02821/) (2022)
- [Variational Boosted Soft Trees](https://mlsanity.com/paper/2302.10706/) (2023)
- [Machine learning's own Industrial Revolution](https://mlsanity.com/paper/2311.02278/) (2023)
- [Academic competitions](https://mlsanity.com/paper/2312.00268/) (2023)
- [Challenge design roadmap](https://mlsanity.com/paper/2401.13693/) (2024)

## Abstract (arXiv)

This paper reports the results and post-challenge analyses of ChaLearn's AutoDL challenge series, which helped sorting out a profusion of AutoML solutions for Deep Learning (DL) that had been introduced in a variety of settings, but lacked fair comparisons. All input data modalities (time series, images, videos, text, tabular) were formatted as tensors and all tasks were multi-label classification problems. Code submissions were executed on hidden tasks, with limited time and computational resources, pushing solutions that get results quickly. In this setting, DL methods dominated, though popular Neural Architecture Search (NAS) was impractical. Solutions relied on fine-tuned pre-trained networks, with architectures matching data modality. Post-challenge tests did not reveal improvements beyond the imposed time limit. While no component is particularly original or novel, a high level modular organization emerged featuring a "meta-learner", "data ingestor", "model selector", "model/learner", and "evaluator". This modularity enabled ablation studies, which revealed the importance of (off-platform) meta-learning, ensembling, and efficient data management. Experiments on heterogeneous module combinations further confirm the (local) optimality of the winning solutions. Our challenge legacy includes an ever-lasting benchmark (http://autodl.chalearn.org), the open-sourced code of the winners, and a free "AutoDL self-service".