# Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search

arXiv: https://arxiv.org/abs/1807.06906 · HTML: https://mlsanity.com/paper/1807.06906/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2018.

## Built on by (most cited first, 30 of 38)

- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [Scaling Laws for Deep Learning](https://mlsanity.com/paper/2108.07686/) (2021)
- [PyGlove: Symbolic Programming for Automated Machine Learning](https://mlsanity.com/paper/2101.08809/) (2021)
- [Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization](https://mlsanity.com/paper/2105.01015/) (2021)
- [Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML](https://mlsanity.com/paper/2303.08485/) (2023)
- [AutoLRS: Automatic Learning-Rate Schedule by Bayesian Optimization on the Fly](https://mlsanity.com/paper/2105.10762/) (2021)
- [Efficient Automated Deep Learning for Time Series Forecasting](https://mlsanity.com/paper/2205.05511/) (2022)
- [Automated Deep Learning: Neural Architecture Search Is Not the End](https://mlsanity.com/paper/2112.09245/) (2021)
- [Neural Architecture Search for Dense Prediction Tasks in Computer Vision](https://mlsanity.com/paper/2202.07242/) (2022)
- [A Survey on Computationally Efficient Neural Architecture Search](https://mlsanity.com/paper/2206.01520/) (2022)
- [Scaling with Collapse: Efficient and Predictable Training of LLM Families](https://mlsanity.com/paper/2509.25087/) (2025)
- [PRE-NAS: Predictor-assisted Evolutionary Neural Architecture Search](https://mlsanity.com/paper/2204.12726/) (2022)
- [A Novel Genetic Algorithm with Hierarchical Evaluation Strategy for Hyperparameter Optimisation of Graph Neural Networks](https://mlsanity.com/paper/2101.09300/) (2021)
- [Adversarial Branch Architecture Search for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2102.06679/) (2021)
- [DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture](https://mlsanity.com/paper/2109.05765/) (2021)
- [DAAS: Differentiable Architecture and Augmentation Policy Search](https://mlsanity.com/paper/2109.15273/) (2021)
- [Towards Less Constrained Macro-Neural Architecture Search](https://mlsanity.com/paper/2203.05508/) (2022)
- [Exploring the Intersection between Neural Architecture Search and Continual Learning](https://mlsanity.com/paper/2206.05625/) (2022)
- [DP-HyPO: An Adaptive Private Hyperparameter Optimization Framework](https://mlsanity.com/paper/2306.05734/) (2023)
- [AutoML for Large Capacity Modeling of Meta's Ranking Systems](https://mlsanity.com/paper/2311.07870/) (2023)
- [Principled Architecture-aware Scaling of Hyperparameters](https://mlsanity.com/paper/2402.17440/) (2024)
- [Accelerating Multi-Objective Neural Architecture Search by Random-Weight Evaluation](https://mlsanity.com/paper/2110.05242/) (2021)
- [AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family](https://mlsanity.com/paper/2111.03555/) (2021)
- [Multi-Agent Automated Machine Learning](https://mlsanity.com/paper/2210.09084/) (2022)
- [Efficient and Joint Hyperparameter and Architecture Search for Collaborative Filtering](https://mlsanity.com/paper/2307.11004/) (2023)
- [Auto-nnU-Net: Towards Automated Medical Image Segmentation](https://mlsanity.com/paper/2505.16561/) (2025)
- [AutoDrop: Training Deep Learning Models with Automatic Learning Rate Drop](https://mlsanity.com/paper/2111.15317/) (2021)
- [Improving Accuracy of Interpretability Measures in Hyperparameter Optimization via Bayesian Algorithm Execution](https://mlsanity.com/paper/2206.05447/) (2022)
- [Automatic Neural Network Hyperparameter Optimization for Extrapolation: Lessons Learned from Visible and Near-Infrared Spectroscopy of Mango Fruit](https://mlsanity.com/paper/2210.01124/) (2022)

## Abstract (arXiv)

While existing work on neural architecture search (NAS) tunes hyperparameters in a separate post-processing step, we demonstrate that architectural choices and other hyperparameter settings interact in a way that can render this separation suboptimal. Likewise, we demonstrate that the common practice of using very few epochs during the main NAS and much larger numbers of epochs during a post-processing step is inefficient due to little correlation in the relative rankings for these two training regimes. To combat both of these problems, we propose to use a recent combination of Bayesian optimization and Hyperband for efficient joint neural architecture and hyperparameter search.