# β-DARTS: Beta-Decay Regularization for Differentiable Architecture Search

arXiv: https://arxiv.org/abs/2203.01665 · HTML: https://mlsanity.com/paper/2203.01665/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 32 references that are in the map.
- First posted on arXiv 3 Mar 2022.

## Builds on (most cited first, 25 of 25)

- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [SNAS: Stochastic Neural Architecture Search](https://mlsanity.com/paper/1812.09926/) (2018)
- [PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search](https://mlsanity.com/paper/1907.05737/) (2019)
- [Searching for A Robust Neural Architecture in Four GPU Hours](https://mlsanity.com/paper/1910.04465/) (2019)
- [Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation](https://mlsanity.com/paper/1904.12760/) (2019)
- [Understanding and Robustifying Differentiable Architecture Search](https://mlsanity.com/paper/1909.09656/) (2019)
- [DARTS+: Improved Differentiable Architecture Search with Early Stopping](https://mlsanity.com/paper/1909.06035/) (2019)
- [Stabilizing Differentiable Architecture Search via Perturbation-based Regularization](https://mlsanity.com/paper/2002.05283/) (2020)
- [Rethinking Architecture Selection in Differentiable NAS](https://mlsanity.com/paper/2108.04392/) (2021)
- [DARTS-: Robustly Stepping out of Performance Collapse Without Indicators](https://mlsanity.com/paper/2009.01027/) (2020)
- [DSNAS: Direct Neural Architecture Search without Parameter Retraining](https://mlsanity.com/paper/2002.09128/) (2020)
- [NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search](https://mlsanity.com/paper/2001.10422/) (2020)
- [iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients](https://mlsanity.com/paper/2106.10784/) (2021)
- [Path-Level Network Transformation for Efficient Architecture Search](https://mlsanity.com/paper/1806.02639/) (2018)
- [Theory-Inspired Path-Regularized Differential Network Architecture Search](https://mlsanity.com/paper/2006.16537/) (2020)
- [DOTS: Decoupling Operation and Topology in Differentiable Architecture Search](https://mlsanity.com/paper/2010.00969/) (2020)
- [DropNAS: Grouped Operation Dropout for Differentiable Architecture Search](https://mlsanity.com/paper/2201.11679/) (2022)
- [MixSearch: Searching for Domain Generalized Medical Image Segmentation Architectures](https://mlsanity.com/paper/2102.13280/) (2021)

## Built on by (most cited first, 30 of 64)

- [EMR-Merging: Tuning-Free High-Performance Model Merging](https://mlsanity.com/paper/2405.17461/) (2024)
- [Can GPT-4 Perform Neural Architecture Search?](https://mlsanity.com/paper/2304.10970/) (2023)
- [ClipSAM: CLIP and SAM Collaboration for Zero-Shot Anomaly Segmentation](https://mlsanity.com/paper/2401.12665/) (2024)
- [DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning](https://mlsanity.com/paper/2509.16105/) (2025)
- [Lambda-DARTS: Mitigating Performance Collapse by Harmonizing Operation Selection among Cells](https://mlsanity.com/paper/2210.07998/) (2022)
- [Generalization Properties of NAS under Activation and Skip Connection Search](https://mlsanity.com/paper/2209.07238/) (2022)
- [Multi-conditioned Graph Diffusion for Neural Architecture Search](https://mlsanity.com/paper/2403.06020/) (2024)
- [LeMo-NADe: Multi-Parameter Neural Architecture Discovery with LLMs](https://mlsanity.com/paper/2402.18443/) (2024)
- [Stimulative Training of Residual Networks: A Social Psychology Perspective of Loafing](https://mlsanity.com/paper/2210.04153/) (2022)
- [Performance-aware Approximation of Global Channel Pruning for Multitask CNNs](https://mlsanity.com/paper/2303.11923/) (2023)
- [Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compression](https://mlsanity.com/paper/2403.15835/) (2024)
- [NADER: Neural Architecture Design via Multi-Agent Collaboration](https://mlsanity.com/paper/2412.19206/) (2024)
- [β-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Search](https://mlsanity.com/paper/2301.06393/) (2023)
- [SEKI: Self-Evolution and Knowledge Inspiration based Neural Architecture Search via Large Language Models](https://mlsanity.com/paper/2502.20422/) (2025)
- [Improving Differentiable Architecture Search via Self-Distillation](https://mlsanity.com/paper/2302.05629/) (2023)
- [Stimulative Training++: Go Beyond The Performance Limits of Residual Networks](https://mlsanity.com/paper/2305.02507/) (2023)
- [Robust NAS under adversarial training: benchmark, theory, and beyond](https://mlsanity.com/paper/2403.13134/) (2024)
- [Efficient Automation of Neural Network Design: A Survey on Differentiable Neural Architecture Search](https://mlsanity.com/paper/2304.05405/) (2023)
- [Insights from the Use of Previously Unseen Neural Architecture Search Datasets](https://mlsanity.com/paper/2404.02189/) (2024)
- [Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis](https://mlsanity.com/paper/2205.05662/) (2022)
- [Robustifying and Boosting Training-Free Neural Architecture Search](https://mlsanity.com/paper/2403.07591/) (2024)
- [OStr-DARTS: Differentiable Neural Architecture Search based on Operation Strength](https://mlsanity.com/paper/2409.14433/) (2024)
- [Subnet-Aware Dynamic Supernet Training for Neural Architecture Search](https://mlsanity.com/paper/2503.10740/) (2025)
- [DCLP: Neural Architecture Predictor with Curriculum Contrastive Learning](https://mlsanity.com/paper/2302.13020/) (2023)
- [Boosting Residual Networks with Group Knowledge](https://mlsanity.com/paper/2308.13772/) (2023)
- [MRF-UNets: Searching UNet with Markov Random Fields](https://mlsanity.com/paper/2207.06168/) (2022)
- [DASS: Differentiable Architecture Search for Sparse neural networks](https://mlsanity.com/paper/2207.06968/) (2022)
- [A2S-NAS: Asymmetric Spectral-Spatial Neural Architecture Search For Hyperspectral Image Classification](https://mlsanity.com/paper/2302.11868/) (2023)
- [Rethink DARTS Search Space and Renovate a New Benchmark](https://mlsanity.com/paper/2306.06852/) (2023)
- [Efficient Architecture Search via Bi-level Data Pruning](https://mlsanity.com/paper/2312.14200/) (2023)

## Abstract (arXiv)

Neural Architecture Search~(NAS) has attracted increasingly more attention in recent years because of its capability to design deep neural networks automatically. Among them, differential NAS approaches such as DARTS, have gained popularity for the search efficiency. However, they suffer from two main issues, the weak robustness to the performance collapse and the poor generalization ability of the searched architectures. To solve these two problems, a simple-but-efficient regularization method, termed as Beta-Decay, is proposed to regularize the DARTS-based NAS searching process. Specifically, Beta-Decay regularization can impose constraints to keep the value and variance of activated architecture parameters from too large. Furthermore, we provide in-depth theoretical analysis on how it works and why it works. Experimental results on NAS-Bench-201 show that our proposed method can help to stabilize the searching process and makes the searched network more transferable across different datasets. In addition, our search scheme shows an outstanding property of being less dependent on training time and data. Comprehensive experiments on a variety of search spaces and datasets validate the effectiveness of the proposed method.