# OODIn: An Optimised On-Device Inference Framework for Heterogeneous Mobile Devices

arXiv: https://arxiv.org/abs/2106.04723 · HTML: https://mlsanity.com/paper/2106.04723/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 8 Jun 2021.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications](https://mlsanity.com/paper/1804.03230/) (2018)
- [SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud](https://mlsanity.com/paper/2008.06402/) (2020)
- [AI Benchmark: All About Deep Learning on Smartphones in 2019](https://mlsanity.com/paper/1910.06663/) (2019)
- [NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision](https://mlsanity.com/paper/1810.10090/) (2018)
- [A First Look at Deep Learning Apps on Smartphones](https://mlsanity.com/paper/1812.05448/) (2018)
- [HAPI: Hardware-Aware Progressive Inference](https://mlsanity.com/paper/2008.03997/) (2020)
- [MobiSR: Efficient On-Device Super-Resolution through Heterogeneous Mobile Processors](https://mlsanity.com/paper/1908.07985/) (2019)
- [EmBench: Quantifying Performance Variations of Deep Neural Networks across Modern Commodity Devices](https://mlsanity.com/paper/1905.07346/) (2019)
- [DynO: Dynamic Onloading of Deep Neural Networks from Cloud to Device](https://mlsanity.com/paper/2104.09949/) (2021)
- [Multi-Exit Semantic Segmentation Networks](https://mlsanity.com/paper/2106.03527/) (2021)
- [Deep Neural Network-based Enhancement for Image and Video Streaming Systems: A Survey and Future Directions](https://mlsanity.com/paper/2106.03727/) (2021)
- [CNNdroid: GPU-Accelerated Execution of Trained Deep Convolutional Neural Networks on Android](https://mlsanity.com/paper/1511.07376/) (2015)

## Built on by (most cited first, 7 of 7)

- [Exploring the Performance and Efficiency of Transformer Models for NLP on Mobile Devices](https://mlsanity.com/paper/2306.11426/) (2023)
- [MultiTASC: A Multi-Tenancy-Aware Scheduler for Cascaded DNN Inference at the Consumer Edge](https://mlsanity.com/paper/2306.12830/) (2023)
- [How to Reach Real-Time AI on Consumer Devices? Solutions for Programmable and Custom Architectures](https://mlsanity.com/paper/2106.15021/) (2021)
- [NAWQ-SR: A Hybrid-Precision NPU Engine for Efficient On-Device Super-Resolution](https://mlsanity.com/paper/2212.09501/) (2022)
- [Mitigating Memory Wall Effects in CNN Engines with On-the-Fly Weights Generation](https://mlsanity.com/paper/2307.13412/) (2023)
- [CARIn: Constraint-Aware and Responsive Inference on Heterogeneous Devices for Single- and Multi-DNN Workloads](https://mlsanity.com/paper/2409.01089/) (2024)
- [MultiTASC++: A Continuously Adaptive Scheduler for Edge-Based Multi-Device Cascade Inference](https://mlsanity.com/paper/2412.04147/) (2024)

## Abstract (arXiv)

Radical progress in the field of deep learning (DL) has led to unprecedented accuracy in diverse inference tasks. As such, deploying DL models across mobile platforms is vital to enable the development and broad availability of the next-generation intelligent apps. Nevertheless, the wide and optimised deployment of DL models is currently hindered by the vast system heterogeneity of mobile devices, the varying computational cost of different DL models and the variability of performance needs across DL applications. This paper proposes OODIn, a framework for the optimised deployment of DL apps across heterogeneous mobile devices. OODIn comprises a novel DL-specific software architecture together with an analytical framework for modelling DL applications that: (1) counteract the variability in device resources and DL models by means of a highly parametrised multi-layer design; and (2) perform a principled optimisation of both model- and system-level parameters through a multi-objective formulation, designed for DL inference apps, in order to adapt the deployment to the user-specified performance requirements and device capabilities. Quantitative evaluation shows that the proposed framework consistently outperforms status-quo designs across heterogeneous devices and delivers up to 4.3x and 3.5x performance gain over highly optimised platform- and model-aware designs respectively, while effectively adapting execution to dynamic changes in resource availability.