# Benchmarking Edge AI Platforms for High-Performance ML Inference

arXiv: https://arxiv.org/abs/2409.14803 · HTML: https://mlsanity.com/paper/2409.14803/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 33 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Builds on (most cited first, 12 of 12)

- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing](https://mlsanity.com/paper/1905.10083/) (2019)
- [Convergence of Edge Computing and Deep Learning: A Comprehensive Survey](https://mlsanity.com/paper/1907.08349/) (2019)
- [Benchmark Analysis of Representative Deep Neural Network Architectures](https://mlsanity.com/paper/1810.00736/) (2018)
- [AI Benchmark: Running Deep Neural Networks on Android Smartphones](https://mlsanity.com/paper/1810.01109/) (2018)
- [Machine learning for Internet of Things data analysis: A survey](https://mlsanity.com/paper/1802.06305/) (2018)
- [DeepEdgeBench: Benchmarking Deep Neural Networks on Edge Devices](https://mlsanity.com/paper/2108.09457/) (2021)
- [EmBench: Quantifying Performance Variations of Deep Neural Networks across Modern Commodity Devices](https://mlsanity.com/paper/1905.07346/) (2019)
- [Latency and Throughput Characterization of Convolutional Neural Networks for Mobile Computer Vision](https://mlsanity.com/paper/1803.09492/) (2018)
- [Edge-Computing-Enabled Smart Cities: A Comprehensive Survey](https://mlsanity.com/paper/1909.08747/) (2019)
- [BLASX: A High Performance Level-3 BLAS Library for Heterogeneous Multi-GPU Computing](https://mlsanity.com/paper/1510.05041/) (2015)

## Built on by (most cited first, 9 of 9)

- [A Persistent-State Dataflow Accelerator for Memory-Bound Linear Attention Decode on FPGA](https://mlsanity.com/paper/2603.05931/) (2026)
- [EdgeWisePersona: A Dataset for On-Device User Profiling from Natural Language Interactions](https://mlsanity.com/paper/2505.11417/) (2025)
- [ConsumerBench: Benchmarking Generative AI Applications on End-User Devices](https://mlsanity.com/paper/2506.17538/) (2025)
- [Edge Artificial Intelligence: A Systematic Review of Evolution, Taxonomic Frameworks, and Future Horizons](https://mlsanity.com/paper/2510.01439/) (2025)
- [Accelerating Mobile Inference through Fine-Grained CPU-GPU Co-Execution](https://mlsanity.com/paper/2510.21081/) (2025)
- [EdgeFlex-Transformer: Transformer Inference for Edge Devices](https://mlsanity.com/paper/2512.19741/) (2025)
- [Context-Driven Performance Modeling for Causal Inference Operators on Neural Processing Units](https://mlsanity.com/paper/2509.25155/) (2025)
- [AI-RAN on NPUs: Baseband Processing Without Baseband Chips](https://mlsanity.com/paper/2607.04224/) (2026)
- [SAT-Edge-Agent: Hardware-in-the-Loop Edge-Agent Orchestration for Onboard Satellite Intelligence](https://mlsanity.com/paper/2608.03728/) (2026)

## Abstract (arXiv)

Edge computing's growing prominence, due to its ability to reduce communication latency and enable real-time processing, is promoting the rise of high-performance, heterogeneous System-on-Chip solutions. While current approaches often involve scaling down modern hardware, the performance characteristics of neural network workloads on these platforms can vary significantly, especially when it comes to parallel processing, which is a critical consideration for edge deployments. To address this, we conduct a comprehensive study comparing the latency and throughput of various linear algebra and neural network inference tasks across CPU-only, CPU/GPU, and CPU/NPU integrated solutions. We find that the Neural Processing Unit (NPU) excels in matrix-vector multiplication (58.6% faster) and some neural network tasks (3.2× faster for video classification and large language models). GPU outperforms in matrix multiplication (22.6% faster) and LSTM networks (2.7× faster) while CPU excels at less parallel operations like dot product. NPU-based inference offers a balance of latency and throughput at lower power consumption. GPU-based inference, though more energy-intensive, performs best with large dimensions and batch sizes. We highlight the potential of heterogeneous computing solutions for edge AI, where diverse compute units can be strategically leveraged to boost accurate and real-time inference.