# Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices

arXiv: https://arxiv.org/abs/2005.05085 · HTML: https://mlsanity.com/paper/2005.05085/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 May 2020.

## Built on by (most cited first, 17 of 17)

- [Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models](https://mlsanity.com/paper/2503.06027/) (2025)
- [Unveiling Energy Efficiency in Deep Learning: Measurement, Prediction, and Scoring across Edge Devices](https://mlsanity.com/paper/2310.18329/) (2023)
- [Benchmarking of DL Libraries and Models on Mobile Devices](https://mlsanity.com/paper/2202.06512/) (2022)
- [Decentralized Low-Latency Collaborative Inference via Ensembles on the Edge](https://mlsanity.com/paper/2206.03165/) (2022)
- [A Survey on Multimodal Wearable Sensor-based Human Action Recognition](https://mlsanity.com/paper/2404.15349/) (2024)
- [Neural Brain: A Neuroscience-inspired Framework for Embodied Agents](https://mlsanity.com/paper/2505.07634/) (2025)
- [Single-Training Collaborative Object Detectors Adaptive to Bandwidth and Computation](https://mlsanity.com/paper/2105.00591/) (2021)
- [ML-EXray: Visibility into ML Deployment on the Edge](https://mlsanity.com/paper/2111.04779/) (2021)
- [LAR-IQA: A Lightweight, Accurate, and Robust No-Reference Image Quality Assessment Model](https://mlsanity.com/paper/2408.17057/) (2024)
- [EPAM: A Predictive Energy Model for Mobile AI](https://mlsanity.com/paper/2303.01509/) (2023)
- [Memory Planning for Deep Neural Networks](https://mlsanity.com/paper/2203.00448/) (2022)
- [Beyond the Model: Data Pre-processing Attack to Deep Learning Models in Android Apps](https://mlsanity.com/paper/2305.03963/) (2023)
- [Robustness Analysis of Deep Learning Frameworks on Mobile Platforms](https://mlsanity.com/paper/2109.09869/) (2021)
- [Studying the Impact of TensorFlow and PyTorch Bindings on Machine Learning Software Quality](https://mlsanity.com/paper/2407.05466/) (2024)
- [Comprehensive Study on Performance Evaluation and Optimization of Model Compression: Bridging Traditional Deep Learning and Large Language Models](https://mlsanity.com/paper/2407.15904/) (2024)
- [SparseByteNN: A Novel Mobile Inference Acceleration Framework Based on Fine-Grained Group Sparsity](https://mlsanity.com/paper/2310.19509/) (2023)
- [Deep Learning Inference on Heterogeneous Mobile Processors: Potentials and Pitfalls](https://mlsanity.com/paper/2405.01851/) (2024)

## Abstract (arXiv)

Due to increasing amounts of data and compute resources, deep learning achieves many successes in various domains. The application of deep learning on the mobile and embedded devices is taken more and more attentions, benchmarking and ranking the AI abilities of mobile and embedded devices becomes an urgent problem to be solved. Considering the model diversity and framework diversity, we propose a benchmark suite, AIoTBench, which focuses on the evaluation of the inference abilities of mobile and embedded devices. AIoTBench covers three typical heavy-weight networks: ResNet50, InceptionV3, DenseNet121, as well as three light-weight networks: SqueezeNet, MobileNetV2, MnasNet. Each network is implemented by three frameworks which are designed for mobile and embedded devices: Tensorflow Lite, Caffe2, Pytorch Mobile. To compare and rank the AI capabilities of the devices, we propose two unified metrics as the AI scores: Valid Images Per Second (VIPS) and Valid FLOPs Per Second (VOPS). Currently, we have compared and ranked 5 mobile devices using our benchmark. This list will be extended and updated soon after.