# DynO: Dynamic Onloading of Deep Neural Networks from Cloud to Device

arXiv: https://arxiv.org/abs/2104.09949 · HTML: https://mlsanity.com/paper/2104.09949/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 68 references that are in the map.
- First posted on arXiv 20 Apr 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks](https://mlsanity.com/paper/1701.05923/) (2017)
- [Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights](https://mlsanity.com/paper/1702.03044/) (2017)
- [Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing](https://mlsanity.com/paper/1910.05316/) (2019)
- [SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud](https://mlsanity.com/paper/2008.06402/) (2020)
- [DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing](https://mlsanity.com/paper/1611.01942/) (2016)
- [AI Benchmark: All About Deep Learning on Smartphones in 2019](https://mlsanity.com/paper/1910.06663/) (2019)
- [Conditional Deep Learning for Energy-Efficient and Enhanced Pattern Recognition](https://mlsanity.com/paper/1509.08971/) (2015)
- [DarkneTZ: Towards Model Privacy at the Edge using Trusted Execution Environments](https://mlsanity.com/paper/2004.05703/) (2020)
- [JALAD: Joint Accuracy- and Latency-Aware Deep Structure Decoupling for Edge-Cloud Execution](https://mlsanity.com/paper/1812.10027/) (2018)
- [Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks](https://mlsanity.com/paper/1705.01626/) (2017)
- [Accelerating Convolutional Neural Networks via Activation Map Compression](https://mlsanity.com/paper/1812.04056/) (2018)
- [HAPI: Hardware-Aware Progressive Inference](https://mlsanity.com/paper/2008.03997/) (2020)
- [EmBench: Quantifying Performance Variations of Deep Neural Networks across Modern Commodity Devices](https://mlsanity.com/paper/1905.07346/) (2019)
- [Trading-off Accuracy and Energy of Deep Inference on Embedded Systems: A Co-Design Approach](https://mlsanity.com/paper/1901.10584/) (2019)
- [CascadeCNN: Pushing the Performance Limits of Quantisation in Convolutional Neural Networks](https://mlsanity.com/paper/1807.05053/) (2018)
- [SETGAN: Scale and Energy Trade-off GANs for Image Applications on Mobile Platforms](https://mlsanity.com/paper/2103.12896/) (2021)

## Built on by (most cited first, 18 of 18)

- [Smart at what cost? Characterising Mobile Deep Neural Networks in the wild](https://mlsanity.com/paper/2109.13963/) (2021)
- [Differentiable Network Pruning for Microcontrollers](https://mlsanity.com/paper/2110.08350/) (2021)
- [FrankenSplit: Efficient Neural Feature Compression with Shallow Variational Bottleneck Injection for Mobile Edge Computing](https://mlsanity.com/paper/2302.10681/) (2023)
- [OODIn: An Optimised On-Device Inference Framework for Heterogeneous Mobile Devices](https://mlsanity.com/paper/2106.04723/) (2021)
- [Accelerating End-Cloud Collaborative Inference via Near Bubble-free Pipeline Optimization](https://mlsanity.com/paper/2501.12388/) (2024)
- [Special Session: Towards an Agile Design Methodology for Efficient, Reliable, and Secure ML Systems](https://mlsanity.com/paper/2204.09514/) (2022)
- [Pex: Memory-efficient Microcontroller Deep Learning through Partial Execution](https://mlsanity.com/paper/2211.17246/) (2022)
- [MultiTASC: A Multi-Tenancy-Aware Scheduler for Cascaded DNN Inference at the Consumer Edge](https://mlsanity.com/paper/2306.12830/) (2023)
- [How to Reach Real-Time AI on Consumer Devices? Solutions for Programmable and Custom Architectures](https://mlsanity.com/paper/2106.15021/) (2021)
- [Serving Graph Neural Networks With Distributed Fog Servers For Smart IoT Services](https://mlsanity.com/paper/2307.01684/) (2023)
- [P/D-Device: Disaggregated Large Language Model between Cloud and Devices](https://mlsanity.com/paper/2508.09035/) (2025)
- [Enabling Cross-Camera Collaboration for Video Analytics on Distributed Smart Cameras](https://mlsanity.com/paper/2401.14132/) (2024)
- [LimitNet: Progressive, Content-Aware Image Offloading for Extremely Weak Devices & Networks](https://mlsanity.com/paper/2504.13736/) (2025)
- [Compressing Many-Shots in In-Context Learning](https://mlsanity.com/paper/2510.16092/) (2025)
- [Fault-Tolerant Collaborative Inference through the Edge-PRUNE Framework](https://mlsanity.com/paper/2206.08152/) (2022)
- [Mondrian: On-Device High-Performance Video Analytics with Compressive Packed Inference](https://mlsanity.com/paper/2403.07598/) (2024)
- [MultiTASC++: A Continuously Adaptive Scheduler for Edge-Based Multi-Device Cascade Inference](https://mlsanity.com/paper/2412.04147/) (2024)
- [MANE: A Multi-Path Adaptive Network for Edge Onloading of Deep Neural Networks](https://mlsanity.com/paper/2609.14660/) (2026)

## Abstract (arXiv)

Recently, there has been an explosive growth of mobile and embedded applications using convolutional neural networks(CNNs). To alleviate their excessive computational demands, developers have traditionally resorted to cloud offloading, inducing high infrastructure costs and a strong dependence on networking conditions. On the other end, the emergence of powerful SoCs is gradually enabling on-device execution. Nonetheless, low- and mid-tier platforms still struggle to run state-of-the-art CNNs sufficiently. In this paper, we present DynO, a distributed inference framework that combines the best of both worlds to address several challenges, such as device heterogeneity, varying bandwidth and multi-objective requirements. Key components that enable this are its novel CNN-specific data packing method, which exploits the variability of precision needs in different parts of the CNN when onloading computation, and its novel scheduler that jointly tunes the partition point and transferred data precision at run time to adapt inference to its execution environment. Quantitative evaluation shows that DynO outperforms the current state-of-the-art, improving throughput by over an order of magnitude over device-only execution and up to 7.9x over competing CNN offloading systems, with up to 60x less data transferred.