# Latency-Aware Differentiable Neural Architecture Search

arXiv: https://arxiv.org/abs/2001.06392 · HTML: https://mlsanity.com/paper/2001.06392/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jan 2020.

## Built on by (most cited first, 10 of 10)

- [A Comprehensive Survey on Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2101.09336/) (2021)
- [HELP: Hardware-Adaptive Efficient Latency Prediction for NAS via Meta-Learning](https://mlsanity.com/paper/2106.08630/) (2021)
- [On Latency Predictors for Neural Architecture Search](https://mlsanity.com/paper/2403.02446/) (2024)
- [MAPLE: Microprocessor A Priori for Latency Estimation](https://mlsanity.com/paper/2111.15106/) (2021)
- [MAPLE-Edge: A Runtime Latency Predictor for Edge Devices](https://mlsanity.com/paper/2204.12950/) (2022)
- [RHNAS: Realizable Hardware and Neural Architecture Search](https://mlsanity.com/paper/2106.09180/) (2021)
- [Rapid Model Architecture Adaption for Meta-Learning](https://mlsanity.com/paper/2109.04925/) (2021)
- [Multi-objective Differentiable Neural Architecture Search](https://mlsanity.com/paper/2402.18213/) (2024)
- [AdaScale: Dynamic Context-aware DNN Scaling via Automated Adaptation Loop on Mobile Devices](https://mlsanity.com/paper/2412.00724/) (2024)
- [MS-DARTS: Mean-Shift Based Differentiable Architecture Search](https://mlsanity.com/paper/2108.09996/) (2021)

## Abstract (arXiv)

Differentiable neural architecture search methods became popular in recent years, mainly due to their low search costs and flexibility in designing the search space. However, these methods suffer the difficulty in optimizing network, so that the searched network is often unfriendly to hardware. This paper deals with this problem by adding a differentiable latency loss term into optimization, so that the search process can tradeoff between accuracy and latency with a balancing coefficient. The core of latency prediction is to encode each network architecture and feed it into a multi-layer regressor, with the training data which can be easily collected from randomly sampling a number of architectures and evaluating them on the hardware. We evaluate our approach on NVIDIA Tesla-P100 GPUs. With 100K sampled architectures (requiring a few hours), the latency prediction module arrives at a relative error of lower than 10%. Equipped with this module, the search method can reduce the latency by 20% meanwhile preserving the accuracy. Our approach also enjoys the ability of being transplanted to a wide range of hardware platforms with very few efforts, or being used to optimizing other non-differentiable factors such as power consumption.