# Regularizing Differentiable Architecture Search with Smooth Activation

arXiv: https://arxiv.org/abs/2504.16306 · HTML: https://mlsanity.com/paper/2504.16306/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 22 Apr 2025.

## Builds on (most cited first, 23 of 23)

- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [SNAS: Stochastic Neural Architecture Search](https://mlsanity.com/paper/1812.09926/) (2018)
- [PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search](https://mlsanity.com/paper/1907.05737/) (2019)
- [Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation](https://mlsanity.com/paper/1904.12760/) (2019)
- [Lightweight Image Super-Resolution with Information Multi-distillation Network](https://mlsanity.com/paper/1909.11856/) (2019)
- [Understanding and Robustifying Differentiable Architecture Search](https://mlsanity.com/paper/1909.09656/) (2019)
- [Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search](https://mlsanity.com/paper/1911.12126/) (2019)
- [FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search](https://mlsanity.com/paper/1907.01845/) (2019)
- [Distilled One-Shot Federated Learning](https://mlsanity.com/paper/2009.07999/) (2020)
- [DARTS+: Improved Differentiable Architecture Search with Early Stopping](https://mlsanity.com/paper/1909.06035/) (2019)
- [Stabilizing Differentiable Architecture Search via Perturbation-based Regularization](https://mlsanity.com/paper/2002.05283/) (2020)
- [Rethinking Architecture Selection in Differentiable NAS](https://mlsanity.com/paper/2108.04392/) (2021)
- [DARTS-: Robustly Stepping out of Performance Collapse Without Indicators](https://mlsanity.com/paper/2009.01027/) (2020)
- [DrNAS: Dirichlet Neural Architecture Search](https://mlsanity.com/paper/2006.10355/) (2020)
- [β-DARTS: Beta-Decay Regularization for Differentiable Architecture Search](https://mlsanity.com/paper/2203.01665/) (2022)
- [Theory-Inspired Path-Regularized Differential Network Architecture Search](https://mlsanity.com/paper/2006.16537/) (2020)
- [Improving Question Generation with Sentence-level Semantic Matching and Answer Position Inferring](https://mlsanity.com/paper/1912.00879/) (2019)
- [SMU: smooth activation function for deep networks using smoothing maximum technique](https://mlsanity.com/paper/2111.04682/) (2021)
- [Adaptive Leader-Follower Formation Control and Obstacle Avoidance via Deep Reinforcement Learning](https://mlsanity.com/paper/1911.06882/) (2019)
- [Asking Complex Questions with Multi-hop Answer-focused Reasoning](https://mlsanity.com/paper/2009.07402/) (2020)
- [Server Averaging for Federated Learning](https://mlsanity.com/paper/2103.11619/) (2021)
- [A Spiking Neural Dynamical Drift-Diffusion Model on Collective Decision Making with Self-Organized Criticality](https://mlsanity.com/paper/1805.02995/) (2018)

## Built on by (most cited first, 1 of 1)

- [FPG-NAS: FLOPs-Aware Gated Differentiable Neural Architecture Search for Efficient 6DoF Pose Estimation](https://mlsanity.com/paper/2508.03618/) (2025)

## Abstract (arXiv)

Differentiable Architecture Search (DARTS) is an efficient Neural Architecture Search (NAS) method but suffers from robustness, generalization, and discrepancy issues. Many efforts have been made towards the performance collapse issue caused by skip dominance with various regularization techniques towards operation weights, path weights, noise injection, and super-network redesign. It had become questionable at a certain point if there could exist a better and more elegant way to retract the search to its intended goal -- NAS is a selection problem. In this paper, we undertake a simple but effective approach, named Smooth Activation DARTS (SA-DARTS), to overcome skip dominance and discretization discrepancy challenges. By leveraging a smooth activation function on architecture weights as an auxiliary loss, our SA-DARTS mitigates the unfair advantage of weight-free operations, converging to fanned-out architecture weight values, and can recover the search process from skip-dominance initialization. Through theoretical and empirical analysis, we demonstrate that the SA-DARTS can yield new state-of-the-art (SOTA) results on NAS-Bench-201, classification, and super-resolution. Further, we show that SA-DARTS can help improve the performance of SOTA models with fewer parameters, such as Information Multi-distillation Network on the super-resolution task.