# Automated Deep Learning: Neural Architecture Search Is Not the End arXiv: https://arxiv.org/abs/2112.09245 ยท HTML: https://mlsanity.com/paper/2112.09245/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Builds on 181 of its 318 references that are in the map. - First posted on arXiv 16 Dec 2021. ## Builds on (most cited first, 30 of 181) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017) - [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015) - [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014) - [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017) - [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015) - [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016) - [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017) - [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018) - [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016) - [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014) - [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016) - [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017) - [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016) - [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018) - [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019) - [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016) - [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016) - [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015) - [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019) - [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018) - [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014) - [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018) - [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016) - [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018) - [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019) ## Built on by (most cited first, 13 of 13) - [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023) - [General Cyclical Training of Neural Networks](https://mlsanity.com/paper/2202.08835/) (2022) - [A Comprehensive Survey on Automated Machine Learning for Recommendations](https://mlsanity.com/paper/2204.01390/) (2022) - [The Roles and Modes of Human Interactions with Automated Machine Learning Systems](https://mlsanity.com/paper/2205.04139/) (2022) - [Principled Architecture-aware Scaling of Hyperparameters](https://mlsanity.com/paper/2402.17440/) (2024) - [Customized Subgraph Selection and Encoding for Drug-drug Interaction Prediction](https://mlsanity.com/paper/2411.01535/) (2024) - [G-Augment: Searching for the Meta-Structure of Data Augmentation Policies for ASR](https://mlsanity.com/paper/2210.10879/) (2022) - [Self-Configuring nnU-Nets Detect Clouds in Satellite Images](https://mlsanity.com/paper/2210.13659/) (2022) - [Applications of Machine Learning in Biopharmaceutical Process Development and Manufacturing: Current Trends, Challenges, and Opportunities](https://mlsanity.com/paper/2310.09991/) (2023) - [Black-box Error Diagnosis in Deep Neural Networks for Computer Vision: a Survey of Tools](https://mlsanity.com/paper/2201.06444/) (2022) - [Evolution Meets Diffusion: Efficient Neural Architecture Generation](https://mlsanity.com/paper/2504.17827/) (2025) - [On Taking Advantage of Opportunistic Meta-knowledge to Reduce Configuration Spaces for Automated Machine Learning](https://mlsanity.com/paper/2208.04376/) (2022) - [Transfer Learning and the Early Estimation of Single-Photon Source Quality using Machine Learning Methods](https://mlsanity.com/paper/2408.11322/) (2024) ## Abstract (arXiv) Deep learning (DL) has proven to be a highly effective approach for developing models in diverse contexts, including visual perception, speech recognition, and machine translation. However, the end-to-end process for applying DL is not trivial. It requires grappling with problem formulation and context understanding, data engineering, model development, deployment, continuous monitoring and maintenance, and so on. Moreover, each of these steps typically relies heavily on humans, in terms of both knowledge and interactions, which impedes the further advancement and democratization of DL. Consequently, in response to these issues, a new field has emerged over the last few years: automated deep learning (AutoDL). This endeavor seeks to minimize the need for human involvement and is best known for its achievements in neural architecture search (NAS), a topic that has been the focus of several surveys. That stated, NAS is not the be-all and end-all of AutoDL. Accordingly, this review adopts an overarching perspective, examining research efforts into automation across the entirety of an archetypal DL workflow. In so doing, this work also proposes a comprehensive set of ten criteria by which to assess existing work in both individual publications and broader research areas. These criteria are: novelty, solution quality, efficiency, stability, interpretability, reproducibility, engineering quality, scalability, generalizability, and eco-friendliness. Thus, ultimately, this review provides an evaluative overview of AutoDL in the early 2020s, identifying where future opportunities for progress may exist.