# Lambda-DARTS: Mitigating Performance Collapse by Harmonizing Operation Selection among Cells

arXiv: https://arxiv.org/abs/2210.07998 · HTML: https://mlsanity.com/paper/2210.07998/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 40 references that are in the map.
- First posted on arXiv 14 Oct 2022.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [Large-Scale Evolution of Image Classifiers](https://mlsanity.com/paper/1703.01041/) (2017)
- [SNAS: Stochastic Neural Architecture Search](https://mlsanity.com/paper/1812.09926/) (2018)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)
- [PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search](https://mlsanity.com/paper/1907.05737/) (2019)
- [Searching for A Robust Neural Architecture in Four GPU Hours](https://mlsanity.com/paper/1910.04465/) (2019)
- [A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions](https://mlsanity.com/paper/2006.02903/) (2020)
- [SMASH: One-Shot Model Architecture Search through HyperNetworks](https://mlsanity.com/paper/1708.05344/) (2017)
- [Understanding and Robustifying Differentiable Architecture Search](https://mlsanity.com/paper/1909.09656/) (2019)
- [On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning](https://mlsanity.com/paper/1704.00805/) (2017)
- [Stabilizing Differentiable Architecture Search via Perturbation-based Regularization](https://mlsanity.com/paper/2002.05283/) (2020)
- [Rethinking Architecture Selection in Differentiable NAS](https://mlsanity.com/paper/2108.04392/) (2021)
- [DARTS-: Robustly Stepping out of Performance Collapse Without Indicators](https://mlsanity.com/paper/2009.01027/) (2020)
- [Residual Connections Encourage Iterative Inference](https://mlsanity.com/paper/1710.04773/) (2017)
- [DrNAS: Dirichlet Neural Architecture Search](https://mlsanity.com/paper/2006.10355/) (2020)
- [DSNAS: Direct Neural Architecture Search without Parameter Retraining](https://mlsanity.com/paper/2002.09128/) (2020)
- [β-DARTS: Beta-Decay Regularization for Differentiable Architecture Search](https://mlsanity.com/paper/2203.01665/) (2022)
- [Highway and Residual Networks learn Unrolled Iterative Estimation](https://mlsanity.com/paper/1612.07771/) (2016)
- [iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients](https://mlsanity.com/paper/2106.10784/) (2021)
- [Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild](https://mlsanity.com/paper/1912.10952/) (2019)
- [Theory-Inspired Path-Regularized Differential Network Architecture Search](https://mlsanity.com/paper/2006.16537/) (2020)
- [DOTS: Decoupling Operation and Topology in Differentiable Architecture Search](https://mlsanity.com/paper/2010.00969/) (2020)

## Built on by (most cited first, 15 of 15)

- [LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimization](https://mlsanity.com/paper/2306.01102/) (2023)
- [Can GPT-4 Perform Neural Architecture Search?](https://mlsanity.com/paper/2304.10970/) (2023)
- [LeMo-NADe: Multi-Parameter Neural Architecture Discovery with LLMs](https://mlsanity.com/paper/2402.18443/) (2024)
- [NADER: Neural Architecture Design via Multi-Agent Collaboration](https://mlsanity.com/paper/2412.19206/) (2024)
- [Architecture-Aware Minimization (A2M): How to Find Flat Minima in Neural Architecture Search](https://mlsanity.com/paper/2503.10404/) (2025)
- [LLM as a Tool, Not an Agent: Code-Mined Tree Transformations for Neural Architecture Search](https://mlsanity.com/paper/2604.16555/) (2026)
- [Multi-objective Differentiable Neural Architecture Search](https://mlsanity.com/paper/2402.18213/) (2024)
- [TopoNAS: Boosting Search Efficiency of Gradient-based NAS via Topological Simplification](https://mlsanity.com/paper/2408.01311/) (2024)
- [EM-DARTS: Hierarchical Differentiable Architecture Search for Eye Movement Recognition](https://mlsanity.com/paper/2409.14432/) (2024)
- [Revisiting Long-Tailed Learning: Insights from an Architectural Perspective](https://mlsanity.com/paper/2411.06098/) (2024)
- [RevoNAD: Reflective Evolutionary Exploration for Neural Architecture Design](https://mlsanity.com/paper/2512.05403/) (2025)
- [Half Search Space is All You Need](https://mlsanity.com/paper/2505.13586/) (2025)
- [confopt: A Library for Implementation and Evaluation of Gradient-based One-Shot NAS Methods](https://mlsanity.com/paper/2507.16533/) (2025)
- [Structuring Open-Ended NAS: Semi-Automated Design Knowledge Structuring with LLMs for Efficient Neural Architecture Search](https://mlsanity.com/paper/2605.19247/) (2026)
- [EvoTreeNAD: Genealogy-Guided Evolution for LLM-Driven Neural Architecture Discovery](https://mlsanity.com/paper/2609.29016/) (2026)

## Abstract (arXiv)

Differentiable neural architecture search (DARTS) is a popular method for neural architecture search (NAS), which performs cell-search and utilizes continuous relaxation to improve the search efficiency via gradient-based optimization. The main shortcoming of DARTS is performance collapse, where the discovered architecture suffers from a pattern of declining quality during search. Performance collapse has become an important topic of research, with many methods trying to solve the issue through either regularization or fundamental changes to DARTS. However, the weight-sharing framework used for cell-search in DARTS and the convergence of architecture parameters has not been analyzed yet. In this paper, we provide a thorough and novel theoretical and empirical analysis on DARTS and its point of convergence. We show that DARTS suffers from a specific structural flaw due to its weight-sharing framework that limits the convergence of DARTS to saturation points of the softmax function. This point of convergence gives an unfair advantage to layers closer to the output in choosing the optimal architecture, causing performance collapse. We then propose two new regularization terms that aim to prevent performance collapse by harmonizing operation selection via aligning gradients of layers. Experimental results on six different search spaces and three different datasets show that our method (Lambda-DARTS) does indeed prevent performance collapse, providing justification for our theoretical analysis and the proposed remedy.