mlsanityData to 7 Oct 2026

A Comprehensive Survey on Hardware-Aware Neural Architecture Search

arXiv 2101.09336 ↗First posted 22 Jan 2021cs.LG cs.CCcited by 60

Not on the map. Why?

Abstract (arXiv)

Neural Architecture Search (NAS) methods have been growing in popularity. These techniques have been fundamental to automate and speed up the time consuming and error-prone process of synthesizing novel Deep Learning (DL) architectures. NAS has been extensively studied in the past few years. Arguably their most significant impact has been in image classification and object detection tasks where the state of the art results have been obtained. Despite the significant success achieved to date, applying NAS to real-world problems still poses significant challenges and is not widely practical. In general, the synthesized Convolution Neural Network (CNN) architectures are too complex to be deployed in resource-limited platforms, such as IoT, mobile, and embedded systems. One solution growing in popularity is to use multi-objective optimization algorithms in the NAS search strategy by taking into account execution latency, energy consumption, memory footprint, etc. This kind of NAS, called hardware-aware NAS (HW-NAS), makes searching the most efficient architecture more complicated and opens several questions. In this survey, we provide a detailed review of existing HW-NAS research and categorize them according to four key dimensions: the search space, the search strategy, the acceleration technique, and the hardware cost estimation strategies. We further discuss the challenges and limitations of existing approaches and potential future directions. This is the first survey paper focusing on hardware-aware NAS. We hope it serves as a valuable reference for the various techniques and algorithms discussed and paves the road for future research towards hardware-aware NAS.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 139 of its 201 references that are in the map. First posted on arXiv 22 Jan 2021.

Builds on

139 of 201 references.

References · 139

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  9. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  11. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  12. 2019Searching for MobileNetV3cited by 1,984
38 more
  1. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  2. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  4. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  5. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  6. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  7. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  8. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  9. 2017Dynamic Routing Between Capsulescited by 689
  10. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  11. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  12. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  13. 2017Progressive Neural Architecture Searchcited by 483
  14. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  15. 2020ResNeSt: Split-Attention Networkscited by 445
  16. 2020NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Searchcited by 412
  17. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  18. 2015Deep Learning with Limited Numerical Precisioncited by 368
  19. 2017Large-Scale Evolution of Image Classifierscited by 348
  20. 2019NAS-Bench-101: Towards Reproducible Neural Architecture Searchcited by 340
  21. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  22. 2019NAS-FPN: Learning Scalable Feature Pyramid Architecture for Object Detectioncited by 289
  23. 2020MCUNet: Tiny Deep Learning on IoT Devicescited by 269
  24. 2019Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentationcited by 258
  25. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  26. 2020Neural Architecture Search without Trainingcited by 241
  27. 2013OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networkscited by 224
  28. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  29. 2018Efficient Multi-objective Neural Architecture Search via Lamarckian Evolutioncited by 151
  30. 2018NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applicationscited by 146
  31. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  32. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  33. 2018Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Searchcited by 122
  34. 2020APQ: Joint Search for Network Architecture, Pruning and Quantization Policycited by 98
  35. 2020NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Sizecited by 98
  36. 2019AutoGAN: Neural Architecture Search for Generative Adversarial Networkscited by 94
  37. 2017MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networkscited by 92
  38. 2019Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hourscited by 87

50 most cited of 139 All 139 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 16; 2024: 10; 2025: 14; 2026: 8.

60 papers.

Sort:

By area:

  1. 2023Neural Architecture Search: Insights from 1000 Paperscited by 136
  2. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  3. 2023Green Edge AI: A Contemporary Surveycited by 24
  4. 2021Multi-objective Asynchronous Successive Halvingcited by 22
  5. 2021Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimizationcited by 19
  6. 2023Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoMLcited by 16
  7. 2022DeepCAVE: An Interactive Analysis Tool for Automated Machine Learningcited by 14
  8. 2023TinyML: Tools, Applications, Challenges, and Future Research Directionscited by 14
  9. 2021A multi-objective perspective on jointly tuning hardware and hyperparameterscited by 12
  10. 2023Hardware-Aware Graph Neural Network Automated Design for Edge Computing Platformscited by 12
  11. 2021FLASH: Fast Neural Architecture Search with Hardware Optimizationcited by 11
  12. 2022Neural Architecture Search for Dense Prediction Tasks in Computer Visioncited by 11
38 more
  1. 2023AnalogNAS: A Neural Network Design Framework for Accurate Inference with Analog In-Memory Computingcited by 10
  2. 2021Measuring what Really Matters: Optimizing Neural Networks for TinyMLcited by 8
  3. 2024On Latency Predictors for Neural Architecture Searchcited by 7
  4. 2022Special Session: Towards an Agile Design Methodology for Efficient, Reliable, and Secure ML Systemscited by 6
  5. 2023When Multi-Task Learning Meets Partial Supervision: A Computer Vision Reviewcited by 6
  6. 2023TFormer: A Transmission-Friendly ViT Model for IoT Devicescited by 5
  7. 2024Graph Neural Networks Automated Design and Deployment on Device-Edge Co-Inference Systemscited by 5
  8. 2024Multi-Objective Hardware Aware Neural Architecture Search using Hardware Cost Diversitycited by 5
  9. 2023SCONNA: A Stochastic Computing Based Optical Accelerator for Ultra-Fast, Energy-Efficient Inference of Integer-Quantized CNNscited by 4
  10. 2025InfiAgent: Self-Evolving Pyramid Agent Framework for Infinite Scenarioscited by 4
  11. 2022Mind the Gap: Measuring Generalization Performance Across Multiple Objectivescited by 3
  12. 2024Embedding Hardware Approximations in Discrete Genetic-based Training for Printed MLPscited by 3
  13. 2025Energy-Efficient Deep Learning for Traffic Classification on Microcontrollerscited by 3
  14. 2023SimQ-NAS: Simultaneous Quantization Policy and Neural Architecture Searchcited by 2
  15. 2025MoENAS: Mixture-of-Expert based Neural Architecture Search for jointly Accurate, Fair, and Robust Edge Deep Neural Networkscited by 2
  16. 2025Fast Data Aware Neural Architecture Search via Supernet Accelerated Evaluationcited by 2
  17. 2025MONAQ: Multi-Objective Neural Architecture Querying for Time-Series Analysis on Resource-Constrained Devicescited by 2
  18. 2025Searching Neural Architectures for Sensor Nodes on IoT Gatewayscited by 2
  19. 2025FPGA-Enabled Machine Learning Applications in Earth Observation: A Systematic Reviewcited by 2
  20. 2025Hardware-Aware Neural Architecture Search for Encrypted Traffic Classification on Resource-Constrained Devicescited by 2
  21. 2025Image Segmentation with Large Language Models: A Survey with Perspectives for Intelligent Transportation Systemscited by 2
  22. 2022EuclidNets: An Alternative Operation for Efficient Inference of Deep Learning Modelscited by 1
  23. 2023End-to-End Neural Network Compression via fracℓ1ℓ2 Regularized Latency Surrogatescited by 1
  24. 2023Sense, Predict, Adapt, Repeat: A Blueprint for Design of New Adaptive AI-Centric Sensing Systemscited by 1
  25. 2023Efficient Verification-Based Face Identificationcited by 1
  26. 2024SONATA: Self-adaptive Evolutionary Framework for Hardware-aware Neural Architecture Searchcited by 1
  27. 2024Combining Neural Architecture Search and Automatic Code Optimization: A Surveycited by 1
  28. 2025LLM-NAS: LLM-driven Hardware-Aware Neural Architecture Searchcited by 1
  29. 2026HAPEns: Hardware-Aware Post-Hoc Ensembling for Tabular Datacited by 1
  30. 2026Deep Learning-Based Anomaly Detection in Spacecraft Telemetry on Edge Devicescited by 1
  31. 2026Hybrid Quantum-Classical Neural Architecture Searchcited by 1
  32. 2022Compilation and Optimizations for Efficient Machine Learning on Embedded Systemscited by 0
  33. 2022SONAR: Joint Architecture and System Optimization Searchcited by 0
  34. 2023Neural Architecture Search for Intel Movidius VPUcited by 0
  35. 2023Grassroots Operator Search for Model Edge Adaptationcited by 0
  36. 2024Towards a tailored mixed-precision sub-8-bit quantization scheme for Gated Recurrent Units using Genetic Algorithmscited by 0
  37. 2024Hardware Aware Ensemble Selection for Balancing Predictive Accuracy and Costcited by 0
  38. 2024HGNAS: Hardware-Aware Graph Neural Architecture Search for Edge Devicescited by 0

50 most cited of 60 All 60 (JSON)