# sharpDARTS: Faster and More Accurate Differentiable Architecture Search

arXiv: https://arxiv.org/abs/1903.09900 · HTML: https://mlsanity.com/paper/1903.09900/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2019.

## Built on by (most cited first, 19 of 19)

- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients](https://mlsanity.com/paper/2301.11300/) (2023)
- [AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search](https://mlsanity.com/paper/2403.19232/) (2024)
- [On Redundancy and Diversity in Cell-based Neural Architecture Search](https://mlsanity.com/paper/2203.08887/) (2022)
- [Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descent](https://mlsanity.com/paper/2409.08770/) (2024)
- [Reconsidering CO2 emissions from Computer Vision](https://mlsanity.com/paper/2104.08702/) (2021)
- [PRE-NAS: Predictor-assisted Evolutionary Neural Architecture Search](https://mlsanity.com/paper/2204.12726/) (2022)
- [Differentiable Architecture Search for Reinforcement Learning](https://mlsanity.com/paper/2106.02229/) (2021)
- [DP-HyPO: An Adaptive Private Hyperparameter Optimization Framework](https://mlsanity.com/paper/2306.05734/) (2023)
- [COVID Detection in Chest CTs: Improving the Baseline on COV19-CT-DB](https://mlsanity.com/paper/2107.04808/) (2021)
- [Single-DARTS: Towards Stable Architecture Search](https://mlsanity.com/paper/2108.08128/) (2021)
- [DASS: Differentiable Architecture Search for Sparse neural networks](https://mlsanity.com/paper/2207.06968/) (2022)
- [Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimization](https://mlsanity.com/paper/2311.08745/) (2023)
- [A Pairwise Comparison Relation-assisted Multi-objective Evolutionary Neural Architecture Search Method with Multi-population Mechanism](https://mlsanity.com/paper/2407.15600/) (2024)
- [Implantable Adaptive Cells: A Novel Enhancement for Pre-Trained U-Nets in Medical Image Segmentation](https://mlsanity.com/paper/2405.03420/) (2024)
- [Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Rate](https://mlsanity.com/paper/2409.09984/) (2024)
- [Warm-starting DARTS using meta-learning](https://mlsanity.com/paper/2205.06355/) (2022)
- [MorphNAS: Differentiable Architecture Search for Morphologically-Aware Multilingual NER](https://mlsanity.com/paper/2508.15836/) (2025)
- [InTrain: Intrinsic Trainability for Zero-Cost Neural Architecture Search](https://mlsanity.com/paper/2606.18676/) (2026)

## Abstract (arXiv)

Neural Architecture Search (NAS) has been a source of dramatic improvements in neural network design, with recent results meeting or exceeding the performance of hand-tuned architectures. However, our understanding of how to represent the search space for neural net architectures and how to search that space efficiently are both still in their infancy. We have performed an in-depth analysis to identify limitations in a widely used search space and a recent architecture search method, Differentiable Architecture Search (DARTS). These findings led us to introduce novel network blocks with a more general, balanced, and consistent design; a better-optimized Cosine Power Annealing learning rate schedule; and other improvements. Our resulting sharpDARTS search is 50% faster with a 20-30% relative improvement in final model error on CIFAR-10 when compared to DARTS. Our best single model run has 1.93% (1.98+/-0.07) validation error on CIFAR-10 and 5.5% error (5.8+/-0.3) on the recently released CIFAR-10.1 test set. To our knowledge, both are state of the art for models of similar size. This model also generalizes competitively to ImageNet at 25.1% top-1 (7.8% top-5) error. We found improvements for existing search spaces but does DARTS generalize to new domains? We propose Differentiable Hyperparameter Grid Search and the HyperCuboid search space, which are representations designed to leverage DARTS for more general parameter optimization. Here we find that DARTS fails to generalize when compared against a human's one shot choice of models. We look back to the DARTS and sharpDARTS search spaces to understand why, and an ablation study reveals an unusual generalization gap. We finally propose Max-W regularization to solve this problem, which proves significantly better than the handmade design. Code will be made available.