# Rethinking Pareto Frontier for Performance Evaluation of Deep Neural Networks

arXiv: https://arxiv.org/abs/2202.09275 · HTML: https://mlsanity.com/paper/2202.09275/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 16 references that are in the map.
- First posted on arXiv 18 Feb 2022.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Hardware as Policy: Mechanical and Computational Co-Optimization using Deep Reinforcement Learning](https://mlsanity.com/paper/2008.04460/) (2020)
- [YOLO Nano: a Highly Compact You Only Look Once Convolutional Neural Network for Object Detection](https://mlsanity.com/paper/1910.01271/) (2019)
- [Deep Learning with Darwin: Evolutionary Synthesis of Deep Neural Networks](https://mlsanity.com/paper/1606.04393/) (2016)

## Built on by (most cited first, 2 of 2)

- [Debiasing Methods for Fairer Neural Models in Vision and Language Research: A Survey](https://mlsanity.com/paper/2211.05617/) (2022)
- [Statistical Hardware Design With Multi-model Active Learning](https://mlsanity.com/paper/2303.08054/) (2023)

## Abstract (arXiv)

Performance optimization of deep learning models is conducted either manually or through automatic architecture search, or a combination of both. On the other hand, their performance strongly depends on the target hardware and how successfully the models were trained. We propose to use a multi-dimensional Pareto frontier to re-define the efficiency measure of candidate deep learning models, where several variables such as training cost, inference latency, and accuracy play a relative role in defining a dominant model. Furthermore, a random version of the multi-dimensional Pareto frontier is introduced to mitigate the uncertainty of accuracy, latency, and throughput of deep learning models in different experimental setups. These two complementary methods can be combined to perform objective benchmarking of deep learning models. Our proposed method is applied to a wide range of deep image classification models trained on ImageNet data. Our method combines competing variables with stochastic nature in a single relative efficiency measure. This allows ranking deep learning models that run efficiently on different hardware, and combining inference efficiency with training efficiency objectively.