# Theory-Inspired Path-Regularized Differential Network Architecture Search

arXiv: https://arxiv.org/abs/2006.16537 · HTML: https://mlsanity.com/paper/2006.16537/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jun 2020.

## Built on by (most cited first, 25 of 25)

- [β-DARTS: Beta-Decay Regularization for Differentiable Architecture Search](https://mlsanity.com/paper/2203.01665/) (2022)
- [AutoBalance: Optimized Loss Functions for Imbalanced Data](https://mlsanity.com/paper/2201.01212/) (2022)
- [ZARTS: On Zero-order Optimization for Neural Architecture Search](https://mlsanity.com/paper/2110.04743/) (2021)
- [On Redundancy and Diversity in Cell-based Neural Architecture Search](https://mlsanity.com/paper/2203.08887/) (2022)
- [Demystifying the Neural Tangent Kernel from a Practical Perspective: Can it be trusted for Neural Architecture Search without training?](https://mlsanity.com/paper/2203.14577/) (2022)
- [Towards Accurate and Compact Architectures via Neural Architecture Transformer](https://mlsanity.com/paper/2102.10301/) (2021)
- [Automated Quantum Circuit Design with Nested Monte Carlo Tree Search](https://mlsanity.com/paper/2207.00132/) (2022)
- [Lambda-DARTS: Mitigating Performance Collapse by Harmonizing Operation Selection among Cells](https://mlsanity.com/paper/2210.07998/) (2022)
- [Generalization Guarantees for Neural Architecture Search with Train-Validation Split](https://mlsanity.com/paper/2104.14132/) (2021)
- [Generalization Properties of NAS under Activation and Skip Connection Search](https://mlsanity.com/paper/2209.07238/) (2022)
- [Automated Graph Machine Learning: Approaches, Libraries, Benchmarks and Directions](https://mlsanity.com/paper/2201.01288/) (2022)
- [A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning](https://mlsanity.com/paper/2106.14749/) (2021)
- [β-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Search](https://mlsanity.com/paper/2301.06393/) (2023)
- [Neural Architecture Search via Combinatorial Multi-Armed Bandit](https://mlsanity.com/paper/2101.00336/) (2021)
- [Efficient Automation of Neural Network Design: A Survey on Differentiable Neural Architecture Search](https://mlsanity.com/paper/2304.05405/) (2023)
- [Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis](https://mlsanity.com/paper/2205.05662/) (2022)
- [AdaSpring: Context-adaptive and Runtime-evolutionary Deep Model Compression for Mobile Applications](https://mlsanity.com/paper/2101.11800/) (2021)
- [DARTS-PRIME: Regularization and Scheduling Improve Constrained Optimization in Differentiable NAS](https://mlsanity.com/paper/2106.11655/) (2021)
- [Efficient Search of Multiple Neural Architectures with Different Complexities via Importance Sampling](https://mlsanity.com/paper/2207.10334/) (2022)
- [A Bilevel Optimization Framework for Imbalanced Data Classification](https://mlsanity.com/paper/2410.11171/) (2024)
- [Regularizing Differentiable Architecture Search with Smooth Activation](https://mlsanity.com/paper/2504.16306/) (2025)
- [DARTS without a Validation Set: Optimizing the Marginal Likelihood](https://mlsanity.com/paper/2112.13023/) (2021)
- [Automating Neural Architecture Design without Search](https://mlsanity.com/paper/2204.11838/) (2022)
- [Automated Search-Space Generation Neural Architecture Search](https://mlsanity.com/paper/2305.18030/) (2023)
- [Sampling Imbalanced Data with Multi-objective Bilevel Optimization](https://mlsanity.com/paper/2506.11315/) (2025)

## Abstract (arXiv)

Despite its high search efficiency, differential architecture search (DARTS) often selects network architectures with dominated skip connections which lead to performance degradation. However, theoretical understandings on this issue remain absent, hindering the development of more advanced methods in a principled way. In this work, we solve this problem by theoretically analyzing the effects of various types of operations, e.g. convolution, skip connection and zero operation, to the network optimization. We prove that the architectures with more skip connections can converge faster than the other candidates, and thus are selected by DARTS. This result, for the first time, theoretically and explicitly reveals the impact of skip connections to fast network optimization and its competitive advantage over other types of operations in DARTS. Then we propose a theory-inspired path-regularized DARTS that consists of two key modules: (i) a differential group-structured sparse binary gate introduced for each operation to avoid unfair competition among operations, and (ii) a path-depth-wise regularization used to incite search exploration for deep architectures that often converge slower than shallow ones as shown in our theory and are not well explored during the search. Experimental results on image classification tasks validate its advantages.