# ANNETTE: Accurate Neural Network Execution Time Estimation with Stacked Models

arXiv: https://arxiv.org/abs/2105.03176 · HTML: https://mlsanity.com/paper/2105.03176/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 37 references that are in the map.
- First posted on arXiv 7 May 2021.

## Builds on (most cited first, 18 of 18)

- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [NAS-Bench-101: Towards Reproducible Neural Architecture Search](https://mlsanity.com/paper/1902.09635/) (2019)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications](https://mlsanity.com/paper/1804.03230/) (2018)
- [Data-free parameter pruning for Deep Neural Networks](https://mlsanity.com/paper/1507.06149/) (2015)
- [Convolutional neural networks with low-rank regularization](https://mlsanity.com/paper/1511.06067/) (2015)
- [Convolutional Neural Networks using Logarithmic Data Representation](https://mlsanity.com/paper/1603.01025/) (2016)
- [Cardiologist-Level Arrhythmia Detection with Convolutional Neural Networks](https://mlsanity.com/paper/1707.01836/) (2017)
- [FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining](https://mlsanity.com/paper/2006.02049/) (2020)
- [SqueezeNAS: Fast neural architecture search for faster semantic segmentation](https://mlsanity.com/paper/1908.01748/) (2019)
- [NeuralPower: Predict and Deploy Energy-Efficient Convolutional Neural Networks](https://mlsanity.com/paper/1710.05420/) (2017)
- [DNN-Chip Predictor: An Analytical Performance Predictor for DNN Accelerators with Various Dataflows and Hardware Architectures](https://mlsanity.com/paper/2002.11270/) (2020)
- [Fixed-Point Performance Analysis of Recurrent Neural Networks](https://mlsanity.com/paper/1512.01322/) (2015)
- [PreVIous: A Methodology for Prediction of Visual Inference Performance on IoT Devices](https://mlsanity.com/paper/1912.06442/) (2019)

## Built on by (most cited first, 1 of 1)

- [Automatic Generation of Fast and Accurate Performance Models for Deep Neural Network Accelerators](https://mlsanity.com/paper/2409.08595/) (2024)

## Abstract (arXiv)

With new accelerator hardware for DNN, the computing power for AI applications has increased rapidly. However, as DNN algorithms become more complex and optimized for specific applications, latency requirements remain challenging, and it is critical to find the optimal points in the design space. To decouple the architectural search from the target hardware, we propose a time estimation framework that allows for modeling the inference latency of DNNs on hardware accelerators based on mapping and layer-wise estimation models. The proposed methodology extracts a set of models from micro-kernel and multi-layer benchmarks and generates a stacked model for mapping and network execution time estimation. We compare estimation accuracy and fidelity of the generated mixed models, statistical models with the roofline model, and a refined roofline model for evaluation. We test the mixed models on the ZCU102 SoC board with DNNDK and Intel Neural Compute Stick 2 on a set of 12 state-of-the-art neural networks. It shows an average estimation error of 3.47% for the DNNDK and 7.44% for the NCS2, outperforming the statistical and analytical layer models for almost all selected networks. For a randomly selected subset of 34 networks of the NASBench dataset, the mixed model reaches fidelity of 0.988 in Spearman's rank correlation coefficient metric. The code of ANNETTE is publicly available at https://github.com/embedded-machine-learning/annette.