# Densely Connected Search Space for More Flexible Neural Architecture Search

arXiv: https://arxiv.org/abs/1906.09607 · HTML: https://mlsanity.com/paper/1906.09607/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jun 2019.

## Built on by (most cited first, 30 of 52)

- [DiNTS: Differentiable Neural Network Topology Search for 3D Medical Image Segmentation](https://mlsanity.com/paper/2103.15954/) (2021)
- [MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens](https://mlsanity.com/paper/2105.15168/) (2021)
- [HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers](https://mlsanity.com/paper/2106.06560/) (2021)
- [CM-NAS: Cross-Modality Neural Architecture Search for Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2101.08467/) (2021)
- [AlphaNet: Improved Training of Supernets with Alpha-Divergence](https://mlsanity.com/paper/2102.07954/) (2021)
- [NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy](https://mlsanity.com/paper/2201.13396/) (2022)
- [Cross-Modal Fine-Tuning: Align then Refine](https://mlsanity.com/paper/2302.05738/) (2023)
- [NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks](https://mlsanity.com/paper/2110.05668/) (2021)
- [Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search](https://mlsanity.com/paper/2101.11342/) (2021)
- [Boosting vision transformers for image retrieval](https://mlsanity.com/paper/2210.11909/) (2022)
- [Rethinking Neural Operations for Diverse Tasks](https://mlsanity.com/paper/2103.15798/) (2021)
- [Efficient Architecture Search for Diverse Tasks](https://mlsanity.com/paper/2204.07554/) (2022)
- [Deep Learning Technique for Human Parsing: A Survey and Outlook](https://mlsanity.com/paper/2301.00394/) (2023)
- [Automated Deep Learning: Neural Architecture Search Is Not the End](https://mlsanity.com/paper/2112.09245/) (2021)
- [Towards Lightweight Super-Resolution with Dual Regression Learning](https://mlsanity.com/paper/2207.07929/) (2022)
- [Gait Recognition in the Wild: A Large-scale Benchmark and NAS-based Baseline](https://mlsanity.com/paper/2205.02692/) (2022)
- [Reconsidering CO2 emissions from Computer Vision](https://mlsanity.com/paper/2104.08702/) (2021)
- [Multi-scale Attentive Image De-raining Networks via Neural Architecture Search](https://mlsanity.com/paper/2207.00728/) (2022)
- [FPUS23: An Ultrasound Fetus Phantom Dataset with Deep Neural Network Evaluations for Fetus Orientations, Fetal Planes, and Anatomical Features](https://mlsanity.com/paper/2303.07852/) (2023)
- [Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization](https://mlsanity.com/paper/2103.02904/) (2021)
- [Neural Architecture Search via Combinatorial Multi-Armed Bandit](https://mlsanity.com/paper/2101.00336/) (2021)
- [Toward Edge-Efficient Dense Predictions with Synergistic Multi-Task Neural Architecture Search](https://mlsanity.com/paper/2210.01384/) (2022)
- [Efficient Automation of Neural Network Design: A Survey on Differentiable Neural Architecture Search](https://mlsanity.com/paper/2304.05405/) (2023)
- [DNArch: Learning Convolutional Neural Architectures by Backpropagation](https://mlsanity.com/paper/2302.05400/) (2023)
- [Neural Fine-Tuning Search for Few-Shot Learning](https://mlsanity.com/paper/2306.09295/) (2023)
- [Boosting Order-Preserving and Transferability for Neural Architecture Search: a Joint Architecture Refined Search and Fine-tuning Approach](https://mlsanity.com/paper/2403.11380/) (2024)
- [Stacked BNAS: Rethinking Broad Convolutional Neural Network for Neural Architecture Search](https://mlsanity.com/paper/2111.07722/) (2021)
- [Adaptive Cross-Layer Attention for Image Restoration](https://mlsanity.com/paper/2203.03619/) (2022)
- [Transferability Estimation Based On Principal Gradient Expectation](https://mlsanity.com/paper/2211.16299/) (2022)
- [Searching Efficient Model-guided Deep Network for Image Denoising](https://mlsanity.com/paper/2104.02525/) (2021)

## Abstract (arXiv)

Neural architecture search (NAS) has dramatically advanced the development of neural network design. We revisit the search space design in most previous NAS methods and find the number and widths of blocks are set manually. However, block counts and block widths determine the network scale (depth and width) and make a great influence on both the accuracy and the model cost (FLOPs/latency). In this paper, we propose to search block counts and block widths by designing a densely connected search space, i.e., DenseNAS. The new search space is represented as a dense super network, which is built upon our designed routing blocks. In the super network, routing blocks are densely connected and we search for the best path between them to derive the final architecture. We further propose a chained cost estimation algorithm to approximate the model cost during the search. Both the accuracy and model cost are optimized in DenseNAS. For experiments on the MobileNetV2-based search space, DenseNAS achieves 75.3% top-1 accuracy on ImageNet with only 361MB FLOPs and 17.9ms latency on a single TITAN-XP. The larger model searched by DenseNAS achieves 76.1% accuracy with only 479M FLOPs. DenseNAS further promotes the ImageNet classification accuracies of ResNet-18, -34 and -50-B by 1.5%, 0.5% and 0.3% with 200M, 600M and 680M FLOPs reduction respectively. The related code is available at https://github.com/JaminFong/DenseNAS.