# ASAP: Architecture Search, Anneal and Prune

arXiv: https://arxiv.org/abs/1904.04123 · HTML: https://mlsanity.com/paper/1904.04123/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2019.

## Built on by (most cited first, 29 of 29)

- [Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond](https://mlsanity.com/paper/2101.11517/) (2021)
- [Searching the Search Space of Vision Transformer](https://mlsanity.com/paper/2111.14725/) (2021)
- [SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning](https://mlsanity.com/paper/2110.11395/) (2021)
- [HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search](https://mlsanity.com/paper/2102.11646/) (2021)
- [Designing the Topology of Graph Neural Networks: A Novel Feature Fusion Perspective](https://mlsanity.com/paper/2112.14531/) (2021)
- [One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking](https://mlsanity.com/paper/2104.00597/) (2021)
- [DropNAS: Grouped Operation Dropout for Differentiable Architecture Search](https://mlsanity.com/paper/2201.11679/) (2022)
- [iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization](https://mlsanity.com/paper/2108.11014/) (2021)
- [Fisher Task Distance and Its Application in Neural Architecture Search](https://mlsanity.com/paper/2103.12827/) (2021)
- [Learning from Mistakes -- A Framework for Neural Architecture Search](https://mlsanity.com/paper/2111.06353/) (2021)
- [Efficient Automation of Neural Network Design: A Survey on Differentiable Neural Architecture Search](https://mlsanity.com/paper/2304.05405/) (2023)
- [A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks](https://mlsanity.com/paper/2101.04243/) (2021)
- [Video Action Recognition Collaborative Learning with Dynamics via PSO-ConvNet Transformer](https://mlsanity.com/paper/2302.09187/) (2023)
- [Learning by Teaching, with Application to Neural Architecture Search](https://mlsanity.com/paper/2103.07009/) (2021)
- [FreeAugment: Data Augmentation Search Across All Degrees of Freedom](https://mlsanity.com/paper/2409.04820/) (2024)
- [Interleaving Learning, with Application to Neural Architecture Search](https://mlsanity.com/paper/2103.07018/) (2021)
- [μDARTS: Model Uncertainty-Aware Differentiable Architecture Search](https://mlsanity.com/paper/2107.11500/) (2021)
- [CONet: Channel Optimization for Convolutional Neural Networks](https://mlsanity.com/paper/2108.06822/) (2021)
- [BINAS: Bilinear Interpretable Neural Architecture Search](https://mlsanity.com/paper/2110.12399/) (2021)
- [DASS: Differentiable Architecture Search for Sparse neural networks](https://mlsanity.com/paper/2207.06968/) (2022)
- [AutoTS: Automatic Time Series Forecasting Model Design Based on Two-Stage Pruning](https://mlsanity.com/paper/2203.14169/) (2022)
- [SpaceEvo: Hardware-Friendly Search Space Design for Efficient INT8 Inference](https://mlsanity.com/paper/2303.08308/) (2023)
- [Graph is all you need? Lightweight data-agnostic neural architecture search without training](https://mlsanity.com/paper/2405.01306/) (2024)
- [Learning from Mistakes based on Class Weighting with Application to Neural Architecture Search](https://mlsanity.com/paper/2112.00275/) (2021)
- [Exploring Complicated Search Spaces with Interleaving-Free Sampling](https://mlsanity.com/paper/2112.02488/) (2021)
- [AutoMC: Automated Model Compression based on Domain Knowledge and Progressive search strategy](https://mlsanity.com/paper/2201.09884/) (2022)
- [Local-to-Global Information Communication for Real-Time Semantic Segmentation Network Search](https://mlsanity.com/paper/2302.08481/) (2023)
- [Degree-Optimized Cumulative Polynomial Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2505.15228/) (2025)
- [EdgeCraft: Automated Model Crafting for Edge IoT](https://mlsanity.com/paper/2609.35167/) (2026)

## Abstract (arXiv)

Automatic methods for Neural Architecture Search (NAS) have been shown to produce state-of-the-art network models. Yet, their main drawback is the computational complexity of the search process. As some primal methods optimized over a discrete search space, thousands of days of GPU were required for convergence. A recent approach is based on constructing a differentiable search space that enables gradient-based optimization, which reduces the search time to a few days. While successful, it still includes some noncontinuous steps, e.g., the pruning of many weak connections at once. In this paper, we propose a differentiable search space that allows the annealing of architecture weights, while gradually pruning inferior operations. In this way, the search converges to a single output network in a continuous manner. Experiments on several vision datasets demonstrate the effectiveness of our method with respect to the search cost and accuracy of the achieved model. Specifically, with 0.2 GPU search days we achieve an error rate of 1.68% on CIFAR-10.