# A Hardware-Aware Framework for Accelerating Neural Architecture Search Across Modalities

arXiv: https://arxiv.org/abs/2205.10358 · HTML: https://mlsanity.com/paper/2205.10358/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 40 references that are in the map.
- First posted on arXiv 19 May 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Single Path One-Shot Neural Architecture Search with Uniform Sampling](https://mlsanity.com/paper/1904.00420/) (2019)
- [A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions](https://mlsanity.com/paper/2006.02903/) (2020)
- [pymoo: Multi-objective Optimization in Python](https://mlsanity.com/paper/2002.04504/) (2020)
- [Hierarchical Representations for Efficient Architecture Search](https://mlsanity.com/paper/1711.00436/) (2017)
- [HAT: Hardware-Aware Transformers for Efficient Natural Language Processing](https://mlsanity.com/paper/2005.14187/) (2020)
- [Neural Collaborative Filtering vs. Matrix Factorization Revisited](https://mlsanity.com/paper/2005.09683/) (2020)
- [NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search](https://mlsanity.com/paper/2007.10396/) (2020)
- [Neural Architecture Transfer](https://mlsanity.com/paper/2005.05859/) (2020)
- [Stronger NAS with Weaker Predictors](https://mlsanity.com/paper/2102.10490/) (2021)
- [HELP: Hardware-Adaptive Efficient Latency Prediction for NAS via Meta-Learning](https://mlsanity.com/paper/2106.08630/) (2021)
- [Two-Archive Evolutionary Algorithm for Constrained Multi-Objective Optimization](https://mlsanity.com/paper/1711.07907/) (2017)
- [Enabling NAS with Automated Super-Network Generation](https://mlsanity.com/paper/2112.10878/) (2021)

## Built on by (most cited first, 6 of 6)

- [LLaMA-NAS: Efficient Neural Architecture Search for Large Language Models](https://mlsanity.com/paper/2405.18377/) (2024)
- [QCQA: Quality and Capacity-aware grouped Query Attention](https://mlsanity.com/paper/2406.10247/) (2024)
- [InstaTune: Instantaneous Neural Architecture Search During Fine-Tuning](https://mlsanity.com/paper/2308.15609/) (2023)
- [Evolution and Efficiency in Neural Architecture Search: Bridging the Gap Between Expert Design and Automated Optimization](https://mlsanity.com/paper/2403.17012/) (2024)
- [SimQ-NAS: Simultaneous Quantization Policy and Neural Architecture Search](https://mlsanity.com/paper/2312.13301/) (2023)
- [CiMNet: Towards Joint Optimization for DNN Architecture and Configuration for Compute-In-Memory Hardware](https://mlsanity.com/paper/2402.11780/) (2024)

## Abstract (arXiv)

Recent advances in Neural Architecture Search (NAS) such as one-shot NAS offer the ability to extract specialized hardware-aware sub-network configurations from a task-specific super-network. While considerable effort has been employed towards improving the first stage, namely, the training of the super-network, the search for derivative high-performing sub-networks is still under-explored. Popular methods decouple the super-network training from the sub-network search and use performance predictors to reduce the computational burden of searching on different hardware platforms. We propose a flexible search framework that automatically and efficiently finds optimal sub-networks that are optimized for different performance metrics and hardware configurations. Specifically, we show how evolutionary algorithms can be paired with lightly trained objective predictors in an iterative cycle to accelerate architecture search in a multi-objective setting for various modalities including machine translation and image classification.