# AtomNAS: Fine-Grained End-to-End Neural Architecture Search

arXiv: https://arxiv.org/abs/1912.09640 · HTML: https://mlsanity.com/paper/1912.09640/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Dec 2019.

## Built on by (most cited first, 30 of 40)

- [Zero-Cost Proxies for Lightweight NAS](https://mlsanity.com/paper/2101.08134/) (2021)
- [RobustART: Benchmarking Robustness on Architecture Design and Training Techniques](https://mlsanity.com/paper/2109.05211/) (2021)
- [AutoCTS: Automated Correlated Time Series Forecasting -- Extended Version](https://mlsanity.com/paper/2112.11174/) (2021)
- [HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers](https://mlsanity.com/paper/2106.06560/) (2021)
- [Teachers Do More Than Teach: Compressing Image-to-Image Models](https://mlsanity.com/paper/2103.03467/) (2021)
- [Contrastive Neural Architecture Search with Neural Architecture Comparators](https://mlsanity.com/paper/2103.05471/) (2021)
- [F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization](https://mlsanity.com/paper/2202.05239/) (2022)
- [Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search](https://mlsanity.com/paper/2101.11342/) (2021)
- [A survey on multi-objective hyperparameter optimization algorithms for Machine Learning](https://mlsanity.com/paper/2111.13755/) (2021)
- [BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer](https://mlsanity.com/paper/2105.08952/) (2021)
- [Rethinking Neural Operations for Diverse Tasks](https://mlsanity.com/paper/2103.15798/) (2021)
- [Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator](https://mlsanity.com/paper/2103.07289/) (2021)
- [NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization](https://mlsanity.com/paper/2104.00031/) (2021)
- [Memory-Efficient Differentiable Transformer Architecture Search](https://mlsanity.com/paper/2105.14669/) (2021)
- [SuperTickets: Drawing Task-Agnostic Lottery Tickets from Supernets via Jointly Architecture Searching and Parameter Pruning](https://mlsanity.com/paper/2207.03677/) (2022)
- [Learning Versatile Neural Architectures by Propagating Network Codes](https://mlsanity.com/paper/2103.13253/) (2021)
- [Reconsidering CO2 emissions from Computer Vision](https://mlsanity.com/paper/2104.08702/) (2021)
- [AdaXpert: Adapting Neural Architecture for Growing Data](https://mlsanity.com/paper/2107.00254/) (2021)
- [Pareto-wise Ranking Classifier for Multi-objective Evolutionary Neural Architecture Search](https://mlsanity.com/paper/2109.07582/) (2021)
- [Neural Architecture Search via Combinatorial Multi-Armed Bandit](https://mlsanity.com/paper/2101.00336/) (2021)
- [Contrastive Self-supervised Neural Architecture Search](https://mlsanity.com/paper/2102.10557/) (2021)
- [Searching for Two-Stream Models in Multivariate Space for Video Recognition](https://mlsanity.com/paper/2108.12957/) (2021)
- [i-Razor: A Differentiable Neural Input Razor for Feature Selection and Dimension Search in DNN-Based Recommender Systems](https://mlsanity.com/paper/2204.00281/) (2022)
- [MathNAS: If Blocks Have a Role in Mathematical Architecture Design](https://mlsanity.com/paper/2311.04943/) (2023)
- [Weight-Entanglement Meets Gradient-Based Neural Architecture Search](https://mlsanity.com/paper/2312.10440/) (2023)
- [MONAS: Efficient Zero-Shot Neural Architecture Search for MCUs](https://mlsanity.com/paper/2408.15034/) (2024)
- [Logic Shrinkage: Learned FPGA Netlist Sparsity for Efficient Neural Network Inference](https://mlsanity.com/paper/2112.02346/) (2021)
- [Automated Dominative Subspace Mining for Efficient Neural Architecture Search](https://mlsanity.com/paper/2210.17180/) (2022)
- [Integrating Large Circular Kernels into CNNs through Neural Architecture Search](https://mlsanity.com/paper/2107.02451/) (2021)
- [CONet: Channel Optimization for Convolutional Neural Networks](https://mlsanity.com/paper/2108.06822/) (2021)

## Abstract (arXiv)

Search space design is very critical to neural architecture search (NAS) algorithms. We propose a fine-grained search space comprised of atomic blocks, a minimal search unit that is much smaller than the ones used in recent NAS algorithms. This search space allows a mix of operations by composing different types of atomic blocks, while the search space in previous methods only allows homogeneous operations. Based on this search space, we propose a resource-aware architecture search framework which automatically assigns the computational resources (e.g., output channel numbers) for each operation by jointly considering the performance and the computational cost. In addition, to accelerate the search process, we propose a dynamic network shrinkage technique which prunes the atomic blocks with negligible influence on outputs on the fly. Instead of a search-and-retrain two-stage paradigm, our method simultaneously searches and trains the target architecture. Our method achieves state-of-the-art performance under several FLOPs configurations on ImageNet with a small searching cost. We open our entire codebase at: https://github.com/meijieru/AtomNAS.