# OmniBoost: Boosting Throughput of Heterogeneous Embedded Devices under Multi-DNN Workload

arXiv: https://arxiv.org/abs/2307.03290 · HTML: https://mlsanity.com/paper/2307.03290/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 6 Jul 2023.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Monte Carlo Tree Search: A Review of Recent Modifications and Applications](https://mlsanity.com/paper/2103.04931/) (2021)
- [High-Throughput CNN Inference on Embedded ARM big.LITTLE Multi-Core Processors](https://mlsanity.com/paper/1903.05898/) (2019)
- [A Survey and Empirical Evaluation of Parallel Deep Learning Frameworks](https://mlsanity.com/paper/2111.04949/) (2021)
- [CNNdroid: GPU-Accelerated Execution of Trained Deep Convolutional Neural Networks on Android](https://mlsanity.com/paper/1511.07376/) (2015)
- [Real-Time Scheduling of Machine Learning Operations on Heterogeneous Neuromorphic SoC](https://mlsanity.com/paper/2209.14777/) (2022)

## Built on by (most cited first, 12 of 12)

- [An LLM-Tool Compiler for Fused Parallel Function Calling](https://mlsanity.com/paper/2405.17438/) (2024)
- [Shared Memory-contention-aware Concurrent DNN Execution for Diversely Heterogeneous System-on-Chips](https://mlsanity.com/paper/2308.05869/) (2023)
- [HiDP: Hierarchical DNN Partitioning for Distributed Inference on Heterogeneous Edge Platforms](https://mlsanity.com/paper/2411.16086/) (2024)
- [RankMap: Priority-Aware Multi-DNN Manager for Heterogeneous Embedded Devices](https://mlsanity.com/paper/2411.17867/) (2024)
- [Ecomap: Sustainability-Driven Optimization of Multi-Tenant DNN Execution on Edge Servers](https://mlsanity.com/paper/2503.04148/) (2025)
- [CARIn: Constraint-Aware and Responsive Inference on Heterogeneous Devices for Single- and Multi-DNN Workloads](https://mlsanity.com/paper/2409.01089/) (2024)
- [Optimizing DNN Inference on Multi-Accelerator SoCs at Training-time](https://mlsanity.com/paper/2409.18566/) (2024)
- [CaMDN: Enhancing Cache Efficiency for Multi-tenant DNNs on Integrated NPUs](https://mlsanity.com/paper/2505.06625/) (2025)
- [Twill: Scheduling Compound AI Systems on Heterogeneous Mobile Edge Platforms](https://mlsanity.com/paper/2507.00491/) (2025)
- [Puzzle: Scheduling Multiple Deep Learning Models on Mobile Device with Heterogeneous Processors](https://mlsanity.com/paper/2508.17764/) (2025)
- [Multi-DNN Inference of Sparse Models on Edge SoCs](https://mlsanity.com/paper/2603.09642/) (2026)
- [TAPAS: Throughput-adaptive Perception for Autonomous Systems](https://mlsanity.com/paper/2607.17317/) (2026)

## Abstract (arXiv)

Modern Deep Neural Networks (DNNs) exhibit profound efficiency and accuracy properties. This has introduced application workloads that comprise of multiple DNN applications, raising new challenges regarding workload distribution. Equipped with a diverse set of accelerators, newer embedded system present architectural heterogeneity, which current run-time controllers are unable to fully utilize. To enable high throughput in multi-DNN workloads, such a controller is ought to explore hundreds of thousands of possible solutions to exploit the underlying heterogeneity. In this paper, we propose OmniBoost, a lightweight and extensible multi-DNN manager for heterogeneous embedded devices. We leverage stochastic space exploration and we combine it with a highly accurate performance estimator to observe a x4.6 average throughput boost compared to other state-of-the-art methods. The evaluation was performed on the HiKey970 development board.