# A Comprehensive Survey on Hardware-Aware Neural Architecture Search

arXiv: https://arxiv.org/abs/2101.09336 · HTML: https://mlsanity.com/paper/2101.09336/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 139 of its 201 references that are in the map.
- First posted on arXiv 22 Jan 2021.

## Builds on (most cited first, 30 of 139)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Efficient Neural Architecture Search via Parameter Sharing](https://mlsanity.com/paper/1802.03268/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Dynamic Routing Between Capsules](https://mlsanity.com/paper/1710.09829/) (2017)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search](https://mlsanity.com/paper/2001.00326/) (2020)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)

## Built on by (most cited first, 30 of 61)

- [Neural Architecture Search: Insights from 1000 Papers](https://mlsanity.com/paper/2301.08727/) (2023)
- [Full Stack Optimization of Transformer Inference: a Survey](https://mlsanity.com/paper/2302.14017/) (2023)
- [Green Edge AI: A Contemporary Survey](https://mlsanity.com/paper/2312.00333/) (2023)
- [Multi-objective Asynchronous Successive Halving](https://mlsanity.com/paper/2106.12639/) (2021)
- [Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization](https://mlsanity.com/paper/2105.01015/) (2021)
- [Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML](https://mlsanity.com/paper/2303.08485/) (2023)
- [DeepCAVE: An Interactive Analysis Tool for Automated Machine Learning](https://mlsanity.com/paper/2206.03493/) (2022)
- [TinyML: Tools, Applications, Challenges, and Future Research Directions](https://mlsanity.com/paper/2303.13569/) (2023)
- [A multi-objective perspective on jointly tuning hardware and hyperparameters](https://mlsanity.com/paper/2106.05680/) (2021)
- [Neural Architecture Search for Dense Prediction Tasks in Computer Vision](https://mlsanity.com/paper/2202.07242/) (2022)
- [Hardware-Aware Graph Neural Network Automated Design for Edge Computing Platforms](https://mlsanity.com/paper/2303.10875/) (2023)
- [FLASH: Fast Neural Architecture Search with Hardware Optimization](https://mlsanity.com/paper/2108.00568/) (2021)
- [AnalogNAS: A Neural Network Design Framework for Accurate Inference with Analog In-Memory Computing](https://mlsanity.com/paper/2305.10459/) (2023)
- [Measuring what Really Matters: Optimizing Neural Networks for TinyML](https://mlsanity.com/paper/2104.10645/) (2021)
- [On Latency Predictors for Neural Architecture Search](https://mlsanity.com/paper/2403.02446/) (2024)
- [Special Session: Towards an Agile Design Methodology for Efficient, Reliable, and Secure ML Systems](https://mlsanity.com/paper/2204.09514/) (2022)
- [When Multi-Task Learning Meets Partial Supervision: A Computer Vision Review](https://mlsanity.com/paper/2307.14382/) (2023)
- [TFormer: A Transmission-Friendly ViT Model for IoT Devices](https://mlsanity.com/paper/2302.07734/) (2023)
- [Graph Neural Networks Automated Design and Deployment on Device-Edge Co-Inference Systems](https://mlsanity.com/paper/2404.05605/) (2024)
- [Multi-Objective Hardware Aware Neural Architecture Search using Hardware Cost Diversity](https://mlsanity.com/paper/2404.12403/) (2024)
- [SCONNA: A Stochastic Computing Based Optical Accelerator for Ultra-Fast, Energy-Efficient Inference of Integer-Quantized CNNs](https://mlsanity.com/paper/2302.07036/) (2023)
- [Energy-Efficient Deep Learning for Traffic Classification on Microcontrollers](https://mlsanity.com/paper/2506.10851/) (2025)
- [InfiAgent: Self-Evolving Pyramid Agent Framework for Infinite Scenarios](https://mlsanity.com/paper/2509.22502/) (2025)
- [Mind the Gap: Measuring Generalization Performance Across Multiple Objectives](https://mlsanity.com/paper/2212.04183/) (2022)
- [Embedding Hardware Approximations in Discrete Genetic-based Training for Printed MLPs](https://mlsanity.com/paper/2402.02930/) (2024)
- [Fast Data Aware Neural Architecture Search via Supernet Accelerated Evaluation](https://mlsanity.com/paper/2502.12690/) (2025)
- [Hardware-Aware Neural Architecture Search for Encrypted Traffic Classification on Resource-Constrained Devices](https://mlsanity.com/paper/2506.11319/) (2025)
- [SimQ-NAS: Simultaneous Quantization Policy and Neural Architecture Search](https://mlsanity.com/paper/2312.13301/) (2023)
- [MoENAS: Mixture-of-Expert based Neural Architecture Search for jointly Accurate, Fair, and Robust Edge Deep Neural Networks](https://mlsanity.com/paper/2502.07422/) (2025)
- [MONAQ: Multi-Objective Neural Architecture Querying for Time-Series Analysis on Resource-Constrained Devices](https://mlsanity.com/paper/2505.10607/) (2025)

## Abstract (arXiv)

Neural Architecture Search (NAS) methods have been growing in popularity. These techniques have been fundamental to automate and speed up the time consuming and error-prone process of synthesizing novel Deep Learning (DL) architectures. NAS has been extensively studied in the past few years. Arguably their most significant impact has been in image classification and object detection tasks where the state of the art results have been obtained. Despite the significant success achieved to date, applying NAS to real-world problems still poses significant challenges and is not widely practical. In general, the synthesized Convolution Neural Network (CNN) architectures are too complex to be deployed in resource-limited platforms, such as IoT, mobile, and embedded systems. One solution growing in popularity is to use multi-objective optimization algorithms in the NAS search strategy by taking into account execution latency, energy consumption, memory footprint, etc. This kind of NAS, called hardware-aware NAS (HW-NAS), makes searching the most efficient architecture more complicated and opens several questions. In this survey, we provide a detailed review of existing HW-NAS research and categorize them according to four key dimensions: the search space, the search strategy, the acceleration technique, and the hardware cost estimation strategies. We further discuss the challenges and limitations of existing approaches and potential future directions. This is the first survey paper focusing on hardware-aware NAS. We hope it serves as a valuable reference for the various techniques and algorithms discussed and paves the road for future research towards hardware-aware NAS.