# High-Throughput CNN Inference on Embedded ARM big.LITTLE Multi-Core Processors

arXiv: https://arxiv.org/abs/1903.05898 · HTML: https://mlsanity.com/paper/1903.05898/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Mar 2019.

## Built on by (most cited first, 19 of 19)

- [Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models](https://mlsanity.com/paper/2503.06027/) (2025)
- [Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inference](https://mlsanity.com/paper/2405.17245/) (2024)
- [OmniBoost: Boosting Throughput of Heterogeneous Embedded Devices under Multi-DNN Workload](https://mlsanity.com/paper/2307.03290/) (2023)
- [Enabling Resource-efficient AIoT System with Cross-level Optimization: A survey](https://mlsanity.com/paper/2309.15467/) (2023)
- [HiDP: Hierarchical DNN Partitioning for Distributed Inference on Heterogeneous Edge Platforms](https://mlsanity.com/paper/2411.16086/) (2024)
- [RankMap: Priority-Aware Multi-DNN Manager for Heterogeneous Embedded Devices](https://mlsanity.com/paper/2411.17867/) (2024)
- [Ecomap: Sustainability-Driven Optimization of Multi-Tenant DNN Execution on Edge Servers](https://mlsanity.com/paper/2503.04148/) (2025)
- [Inference Latency Prediction at the Edge](https://mlsanity.com/paper/2210.02620/) (2022)
- [The Effects of Partitioning Strategies on Energy Consumption in Distributed CNN Inference at The Edge](https://mlsanity.com/paper/2210.08392/) (2022)
- [CARIn: Constraint-Aware and Responsive Inference on Heterogeneous Devices for Single- and Multi-DNN Workloads](https://mlsanity.com/paper/2409.01089/) (2024)
- [Optimizing Multi-DNN Inference on Mobile Devices through Heterogeneous Processor Co-Execution](https://mlsanity.com/paper/2503.21109/) (2025)
- [Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism](https://mlsanity.com/paper/2505.23219/) (2025)
- [Collate: Collaborative Neural Network Learning for Latency-Critical Edge Systems](https://mlsanity.com/paper/2607.08013/) (2026)
- [Shisha: Online scheduling of CNN pipelines on heterogeneous architectures](https://mlsanity.com/paper/2202.11575/) (2022)
- [DLAS: An Exploration and Assessment of the Deep Learning Acceleration Stack](https://mlsanity.com/paper/2311.08909/) (2023)
- [A Framework to Enable Algorithmic Design Choice Exploration in DNNs](https://mlsanity.com/paper/2410.08300/) (2024)
- [Twill: Scheduling Compound AI Systems on Heterogeneous Mobile Edge Platforms](https://mlsanity.com/paper/2507.00491/) (2025)
- [Power-Performance Characterization of TinyML Systems](https://mlsanity.com/paper/2608.21646/) (2026)
- [Para-Pipe: Exploiting Hierarchical Operator Parallelism of ML Computational Graphs on SoCs](https://mlsanity.com/paper/2609.04168/) (2026)

## Abstract (arXiv)

IoT Edge intelligence requires Convolutional Neural Network (CNN) inference to take place in the edge devices itself. ARM big.LITTLE architecture is at the heart of prevalent commercial edge devices. It comprises of single-ISA heterogeneous cores grouped into multiple homogeneous clusters that enable power and performance trade-offs. All cores are expected to be simultaneously employed in inference to attain maximal throughput. However, high communication overhead involved in parallelization of computations from convolution kernels across clusters is detrimental to throughput. We present an alternative framework called Pipe-it that employs pipelined design to split convolutional layers across clusters while limiting parallelization of their respective kernels to the assigned cluster. We develop a performance-prediction model that utilizes only the convolutional layer descriptors to predict the execution time of each layer individually on all permitted core configurations (type and count). Pipe-it then exploits the predictions to create a balanced pipeline using an efficient design space exploration algorithm. Pipe-it on average results in a 39% higher throughput than the highest antecedent throughput.