# Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild

arXiv: https://arxiv.org/abs/1912.10952 · HTML: https://mlsanity.com/paper/1912.10952/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2019.

## Built on by (most cited first, 30 of 40)

- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition](https://mlsanity.com/paper/2102.01063/) (2021)
- [ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models](https://mlsanity.com/paper/2403.16187/) (2024)
- [APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models](https://mlsanity.com/paper/2402.14866/) (2024)
- [Designing the Topology of Graph Neural Networks: A Novel Feature Fusion Perspective](https://mlsanity.com/paper/2112.14531/) (2021)
- [Trilevel Neural Architecture Search for Efficient Single Image Super-Resolution](https://mlsanity.com/paper/2101.06658/) (2021)
- [FBNetV5: Neural Architecture Search for Multiple Tasks in One Run](https://mlsanity.com/paper/2111.10007/) (2021)
- [Lambda-DARTS: Mitigating Performance Collapse by Harmonizing Operation Selection among Cells](https://mlsanity.com/paper/2210.07998/) (2022)
- [Efficient Joint-Dimensional Search with Solution Space Regularization for Real-Time Semantic Segmentation](https://mlsanity.com/paper/2208.05271/) (2022)
- [Learning from Mistakes -- A Framework for Neural Architecture Search](https://mlsanity.com/paper/2111.06353/) (2021)
- [Improving Differentiable Architecture Search via Self-Distillation](https://mlsanity.com/paper/2302.05629/) (2023)
- [OTOv3: Automatic Architecture-Agnostic Neural Network Training and Compression from Structured Pruning to Erasing Operators](https://mlsanity.com/paper/2312.09411/) (2023)
- [Contrastive Self-supervised Neural Architecture Search](https://mlsanity.com/paper/2102.10557/) (2021)
- [NAS-LID: Efficient Neural Architecture Search with Local Intrinsic Dimension](https://mlsanity.com/paper/2211.12759/) (2022)
- [CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Framework](https://mlsanity.com/paper/2406.01414/) (2024)
- [NAX: Co-Designing Neural Network and Hardware Architecture for Memristive Xbar based Computing Systems](https://mlsanity.com/paper/2106.12125/) (2021)
- [CHASE: Robust Visual Tracking via Cell-Level Differentiable Neural Architecture Search](https://mlsanity.com/paper/2107.03463/) (2021)
- [CR-LSO: Convex Neural Architecture Optimization in the Latent Space of Graph Variational Autoencoder with Input Convex Neural Networks](https://mlsanity.com/paper/2211.05950/) (2022)
- [A lightweight network for photovoltaic cell defect detection in electroluminescence images based on neural architecture search and knowledge distillation](https://mlsanity.com/paper/2302.07455/) (2023)
- [Carbon-Efficient Neural Architecture Search](https://mlsanity.com/paper/2307.04131/) (2023)
- [Shrink-Perturb Improves Architecture Mixing during Population Based Training for Neural Architecture Search](https://mlsanity.com/paper/2307.15621/) (2023)
- [One Network Doesn't Rule Them All: Moving Beyond Handcrafted Architectures in Self-Supervised Learning](https://mlsanity.com/paper/2203.08130/) (2022)
- [NASiam: Efficient Representation Learning using Neural Architecture Search for Siamese Networks](https://mlsanity.com/paper/2302.00059/) (2023)
- [Improving Prompt Tuning with Learned Prompting Layers](https://mlsanity.com/paper/2310.20127/) (2023)
- [Multi-Objective Neural Architecture Search by Learning Search Space Partitions](https://mlsanity.com/paper/2406.00291/) (2024)
- [ZO-DARTS++: An Efficient and Size-Variable Zeroth-Order Neural Architecture Search Algorithm](https://mlsanity.com/paper/2503.06092/) (2025)
- [DDS-NAS: Dynamic Data Selection within Neural Architecture Search via On-line Hard Example Mining applied to Image Classification](https://mlsanity.com/paper/2506.14667/) (2025)
- [Zero-Shot Neural Architecture Search with Weighted Response Correlation](https://mlsanity.com/paper/2507.08841/) (2025)
- [D-DARTS: Distributed Differentiable Architecture Search](https://mlsanity.com/paper/2108.09306/) (2021)
- [Efficient Re-parameterization Operations Search for Easy-to-Deploy Network Based on Directional Evolutionary Strategy](https://mlsanity.com/paper/2204.06403/) (2022)

## Abstract (arXiv)

With the rapid development of neural architecture search (NAS), researchers found powerful network architectures for a wide range of vision tasks. However, it remains unclear if the searched architecture can transfer across different types of tasks as manually designed ones did. This paper puts forward this problem, referred to as NAS in the wild, which explores the possibility of finding the optimal architecture in a proxy dataset and then deploying it to mostly unseen scenarios. We instantiate this setting using a currently popular algorithm named differentiable architecture search (DARTS), which often suffers unsatisfying performance while being transferred across different tasks. We argue that the accuracy drop originates from the formulation that uses a super-network for search but a sub-network for re-training. The different properties of these stages have resulted in a significant optimization gap, and consequently, the architectural parameters "over-fit" the super-network. To alleviate the gap, we present a progressive method that gradually increases the network depth during the search stage, which leads to the Progressive DARTS (P-DARTS) algorithm. With a reduced search cost (7 hours on a single GPU), P-DARTS achieves improved performance on both the proxy dataset (CIFAR10) and a few target problems (ImageNet classification, COCO detection and three ReID benchmarks). Our code is available at urlhttps://github.com/chenxin061/pdarts.