# DSNAS: Direct Neural Architecture Search without Parameter Retraining

arXiv: https://arxiv.org/abs/2002.09128 · HTML: https://mlsanity.com/paper/2002.09128/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2020.

## Built on by (most cited first, 30 of 69)

- [Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond](https://mlsanity.com/paper/2101.11517/) (2021)
- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [Can GPT-4 Perform Neural Architecture Search?](https://mlsanity.com/paper/2304.10970/) (2023)
- [Exploring Relational Context for Multi-Task Dense Prediction](https://mlsanity.com/paper/2104.13874/) (2021)
- [β-DARTS: Beta-Decay Regularization for Differentiable Architecture Search](https://mlsanity.com/paper/2203.01665/) (2022)
- [Neural Architecture Search for Spiking Neural Networks](https://mlsanity.com/paper/2201.10355/) (2022)
- [DiNTS: Differentiable Neural Network Topology Search for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.15954/) (2021)
- [NAS-OoD: Neural Architecture Search for Out-of-Distribution Generalization](https://mlsanity.com/paper/2109.02038/) (2021)
- [Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search](https://mlsanity.com/paper/2101.11342/) (2021)
- [Generalizing Few-Shot NAS with Gradient Matching](https://mlsanity.com/paper/2203.15207/) (2022)
- [Linear, or Non-Linear, That is the Question!](https://mlsanity.com/paper/2111.07265/) (2021)
- [DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks](https://mlsanity.com/paper/2206.00843/) (2022)
- [Recent Progress in the CUHK Dysarthric Speech Recognition System](https://mlsanity.com/paper/2201.05845/) (2022)
- [Automatic Mixed-Precision Quantization Search of BERT](https://mlsanity.com/paper/2112.14938/) (2021)
- [EZNAS: Evolving Zero Cost Proxies For Neural Architecture Scoring](https://mlsanity.com/paper/2209.07413/) (2022)
- [Lambda-DARTS: Mitigating Performance Collapse by Harmonizing Operation Selection among Cells](https://mlsanity.com/paper/2210.07998/) (2022)
- [Memory-Efficient Differentiable Transformer Architecture Search](https://mlsanity.com/paper/2105.14669/) (2021)
- [SPIDER: Searching Personalized Neural Architecture for Federated Learning](https://mlsanity.com/paper/2112.13939/) (2021)
- [Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators](https://mlsanity.com/paper/2106.06575/) (2021)
- [RepNAS: Searching for Efficient Re-parameterizing Blocks](https://mlsanity.com/paper/2109.03508/) (2021)
- [LeMo-NADe: Multi-Parameter Neural Architecture Discovery with LLMs](https://mlsanity.com/paper/2402.18443/) (2024)
- [iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization](https://mlsanity.com/paper/2108.11014/) (2021)
- [UDC: Unified DNAS for Compressible TinyML Models](https://mlsanity.com/paper/2201.05842/) (2022)
- [EEEA-Net: An Early Exit Evolutionary Neural Architecture Search](https://mlsanity.com/paper/2108.06156/) (2021)
- [NADER: Neural Architecture Design via Multi-Agent Collaboration](https://mlsanity.com/paper/2412.19206/) (2024)
- [Pareto-wise Ranking Classifier for Multi-objective Evolutionary Neural Architecture Search](https://mlsanity.com/paper/2109.07582/) (2021)
- [β-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Search](https://mlsanity.com/paper/2301.06393/) (2023)
- [RD-NAS: Enhancing One-shot Supernet Ranking Ability via Ranking Distillation from Zero-cost Proxies](https://mlsanity.com/paper/2301.09850/) (2023)
- [MetaMix: Meta-state Precision Searcher for Mixed-precision Activation Quantization](https://mlsanity.com/paper/2311.06798/) (2023)
- [SEKI: Self-Evolution and Knowledge Inspiration based Neural Architecture Search via Large Language Models](https://mlsanity.com/paper/2502.20422/) (2025)

## Abstract (arXiv)

If NAS methods are solutions, what is the problem? Most existing NAS methods require two-stage parameter optimization. However, performance of the same architecture in the two stages correlates poorly. In this work, we propose a new problem definition for NAS, task-specific end-to-end, based on this observation. We argue that given a computer vision task for which a NAS method is expected, this definition can reduce the vaguely-defined NAS evaluation to i) accuracy of this task and ii) the total computation consumed to finally obtain a model with satisfying accuracy. Seeing that most existing methods do not solve this problem directly, we propose DSNAS, an efficient differentiable NAS framework that simultaneously optimizes architecture and parameters with a low-biased Monte Carlo estimate. Child networks derived from DSNAS can be deployed directly without parameter retraining. Comparing with two-stage methods, DSNAS successfully discovers networks with comparable accuracy (74.4%) on ImageNet in 420 GPU hours, reducing the total time by more than 34%. Our implementation is available at https://github.com/SNAS-Series/SNAS-Series.