# Stabilizing Differentiable Architecture Search via Perturbation-based Regularization

arXiv: https://arxiv.org/abs/2002.05283 · HTML: https://mlsanity.com/paper/2002.05283/

## Key facts

- Cited by 102 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2020.

## Built on by (most cited first, 30 of 102)

- [Symbolic Discovery of Optimization Algorithms](https://mlsanity.com/paper/2302.06675/) (2023)
- [When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations](https://mlsanity.com/paper/2106.01548/) (2021)
- [Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective](https://mlsanity.com/paper/2102.11535/) (2021)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2111.14213/) (2021)
- [Rethinking Architecture Selection in Differentiable NAS](https://mlsanity.com/paper/2108.04392/) (2021)
- [Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks](https://mlsanity.com/paper/2110.03825/) (2021)
- [β-DARTS: Beta-Decay Regularization for Differentiable Architecture Search](https://mlsanity.com/paper/2203.01665/) (2022)
- [ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients](https://mlsanity.com/paper/2301.11300/) (2023)
- [iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients](https://mlsanity.com/paper/2106.10784/) (2021)
- [ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models](https://mlsanity.com/paper/2403.16187/) (2024)
- [NIPQ: Noise proxy-based Integrated Pseudo-Quantization](https://mlsanity.com/paper/2206.00820/) (2022)
- [AdvRush: Searching for Adversarially Robust Neural Architectures](https://mlsanity.com/paper/2108.01289/) (2021)
- [NASI: Label- and Data-agnostic Neural Architecture Search at Initialization](https://mlsanity.com/paper/2109.00817/) (2021)
- [ASP: Learn a Universal Neural Solver!](https://mlsanity.com/paper/2303.00466/) (2023)
- [DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning](https://mlsanity.com/paper/2509.16105/) (2025)
- [Generalizing Few-Shot NAS with Gradient Matching](https://mlsanity.com/paper/2203.15207/) (2022)
- [Shapley-NAS: Discovering Operation Contribution for Neural Architecture Search](https://mlsanity.com/paper/2206.09811/) (2022)
- [CATE: Computation-aware Neural Architecture Encoding with Transformers](https://mlsanity.com/paper/2102.07108/) (2021)
- [Meta Navigator: Search for a Good Adaptation Policy for Few-shot Learning](https://mlsanity.com/paper/2109.05749/) (2021)
- [ZARTS: On Zero-order Optimization for Neural Architecture Search](https://mlsanity.com/paper/2110.04743/) (2021)
- [Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?](https://mlsanity.com/paper/2203.14577/) (2022)
- [Towards Accurate and Compact Architectures via Neural Architecture Transformer](https://mlsanity.com/paper/2102.10301/) (2021)
- [Unifying and Boosting Gradient-Based Training-Free Neural Architecture Search](https://mlsanity.com/paper/2201.09785/) (2022)
- [Lambda-DARTS: Mitigating Performance Collapse by Harmonizing Operation Selection among Cells](https://mlsanity.com/paper/2210.07998/) (2022)
- [PTP: Boosting Stability and Performance of Prompt Tuning with Perturbation-Based Regularizer](https://mlsanity.com/paper/2305.02423/) (2023)
- [Accelerating Neural Architecture Search via Proxy Data](https://mlsanity.com/paper/2106.04784/) (2021)
- [Neural Architecture Search for Dense Prediction Tasks in Computer Vision](https://mlsanity.com/paper/2202.07242/) (2022)
- [Towards Multi-modal Transformers in Federated Learning](https://mlsanity.com/paper/2404.12467/) (2024)
- [iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization](https://mlsanity.com/paper/2108.11014/) (2021)

## Abstract (arXiv)

Differentiable architecture search (DARTS) is a prevailing NAS solution to identify architectures. Based on the continuous relaxation of the architecture space, DARTS learns a differentiable architecture weight and largely reduces the search cost. However, its stability has been challenged for yielding deteriorating architectures as the search proceeds. We find that the precipitous validation loss landscape, which leads to a dramatic performance drop when distilling the final architecture, is an essential factor that causes instability. Based on this observation, we propose a perturbation-based regularization - SmoothDARTS (SDARTS), to smooth the loss landscape and improve the generalizability of DARTS-based methods. In particular, our new formulations stabilize DARTS-based methods by either random smoothing or adversarial attack. The search trajectory on NAS-Bench-1Shot1 demonstrates the effectiveness of our approach and due to the improved stability, we achieve performance gain across various search spaces on 4 datasets. Furthermore, we mathematically show that SDARTS implicitly regularizes the Hessian norm of the validation loss, which accounts for a smoother loss landscape and improved performance.