# Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation

arXiv: https://arxiv.org/abs/1904.12760 · HTML: https://mlsanity.com/paper/1904.12760/

## Key facts

- Cited by 230 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2019.

## Built on by (most cited first, 30 of 230)

- [Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective](https://mlsanity.com/paper/2102.11535/) (2021)
- [Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond](https://mlsanity.com/paper/2101.11517/) (2021)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [Rethinking Architecture Selection in Differentiable NAS](https://mlsanity.com/paper/2108.04392/) (2021)
- [LightTrack: Finding Lightweight Neural Networks for Object Tracking via One-Shot Architecture Search](https://mlsanity.com/paper/2104.14545/) (2021)
- [DC-BENCH: Dataset Condensation Benchmark](https://mlsanity.com/paper/2207.09639/) (2022)
- [β-DARTS: Beta-Decay Regularization for Differentiable Architecture Search](https://mlsanity.com/paper/2203.01665/) (2022)
- [Towards Gradient-based Bilevel Optimization with Non-convex Followers and Beyond](https://mlsanity.com/paper/2110.00455/) (2021)
- [A Value-Function-based Interior-point Method for Non-convex Bi-level Optimization](https://mlsanity.com/paper/2106.07991/) (2021)
- [ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients](https://mlsanity.com/paper/2301.11300/) (2023)
- [iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients](https://mlsanity.com/paper/2106.10784/) (2021)
- [DiNTS: Differentiable Neural Network Topology Search for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.15954/) (2021)
- [Searching the Search Space of Vision Transformer](https://mlsanity.com/paper/2111.14725/) (2021)
- [CM-NAS: Cross-Modality Neural Architecture Search for Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2101.08467/) (2021)
- [Teachers Do More Than Teach: Compressing Image-to-Image Models](https://mlsanity.com/paper/2103.03467/) (2021)
- [PocketNet: Extreme Lightweight Face Recognition Network using Neural Architecture Search and Multi-Step Knowledge Distillation](https://mlsanity.com/paper/2108.10710/) (2021)
- [AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks](https://mlsanity.com/paper/2306.08107/) (2023)
- [Contrastive Neural Architecture Search with Neural Architecture Comparators](https://mlsanity.com/paper/2103.05471/) (2021)
- [Visual Tuning](https://mlsanity.com/paper/2305.06061/) (2023)
- [Neural Architecture Search with Random Labels](https://mlsanity.com/paper/2101.11834/) (2021)
- [Stronger NAS with Weaker Predictors](https://mlsanity.com/paper/2102.10490/) (2021)
- [AdvRush: Searching for Adversarially Robust Neural Architectures](https://mlsanity.com/paper/2108.01289/) (2021)
- [NASI: Label- and Data-agnostic Neural Architecture Search at Initialization](https://mlsanity.com/paper/2109.00817/) (2021)
- [Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search](https://mlsanity.com/paper/2101.11342/) (2021)
- [Averaged Method of Multipliers for Bi-Level Optimization without Lower-Level Strong Convexity](https://mlsanity.com/paper/2302.03407/) (2023)
- [Generalizing Few-Shot NAS with Gradient Matching](https://mlsanity.com/paper/2203.15207/) (2022)
- [Value-Function-based Sequential Minimization for Bi-level Optimization](https://mlsanity.com/paper/2110.04974/) (2021)
- [Core-set Sampling for Efficient Neural Architecture Search](https://mlsanity.com/paper/2107.06869/) (2021)
- [Shapley-NAS: Discovering Operation Contribution for Neural Architecture Search](https://mlsanity.com/paper/2206.09811/) (2022)
- [Auto-HeG: Automated Graph Neural Network on Heterophilic Graphs](https://mlsanity.com/paper/2302.12357/) (2023)

## Abstract (arXiv)

Recently, differentiable search methods have made major progress in reducing the computational costs of neural architecture search. However, these approaches often report lower accuracy in evaluating the searched architecture or transferring it to another dataset. This is arguably due to the large gap between the architecture depths in search and evaluation scenarios. In this paper, we present an efficient algorithm which allows the depth of searched architectures to grow gradually during the training procedure. This brings two issues, namely, heavier computational overheads and weaker search stability, which we solve using search space approximation and regularization, respectively. With a significantly reduced search time (~7 hours on a single GPU), our approach achieves state-of-the-art performance on both the proxy dataset (CIFAR10 or CIFAR100) and the target dataset (ImageNet). Code is available at https://github.com/chenxin061/pdarts.