# NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision

arXiv: https://arxiv.org/abs/1810.10090 · HTML: https://mlsanity.com/paper/1810.10090/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2018.

## Built on by (most cited first, 30 of 68)

- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout](https://mlsanity.com/paper/2102.13451/) (2021)
- [Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions](https://mlsanity.com/paper/2106.05022/) (2021)
- [LLMCad: Fast and Scalable On-device Large Language Model Inference](https://mlsanity.com/paper/2309.04255/) (2023)
- [Federated Learning for Computationally-Constrained Heterogeneous Devices: A Survey](https://mlsanity.com/paper/2307.09182/) (2023)
- [SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget](https://mlsanity.com/paper/2308.15030/) (2023)
- [A Survey on Collaborative DNN Inference for Edge Intelligence](https://mlsanity.com/paper/2207.07812/) (2022)
- [LegoDNN: Block-grained Scaling of Deep Neural Networks for Mobile Vision](https://mlsanity.com/paper/2112.09852/) (2021)
- [On-device Training: A First Overview on Existing Systems](https://mlsanity.com/paper/2212.00824/) (2022)
- [AdaptiveNet: Post-deployment Neural Architecture Adaptation for Diverse Edge Environments](https://mlsanity.com/paper/2303.07129/) (2023)
- [Multi-Exit Semantic Segmentation Networks](https://mlsanity.com/paper/2106.03527/) (2021)
- [Real-time Neural Network Inference on Extremely Weak Devices: Agile Offloading with Explainable AI](https://mlsanity.com/paper/2312.14229/) (2023)
- [STI: Turbocharge NLP Inference at the Edge via Elastic Pipelining](https://mlsanity.com/paper/2207.05022/) (2022)
- [HydraViT: Stacking Heads for a Scalable ViT](https://mlsanity.com/paper/2409.17978/) (2024)
- [Deep Neural Network-based Enhancement for Image and Video Streaming Systems: A Survey and Future Directions](https://mlsanity.com/paper/2106.03727/) (2021)
- [Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques](https://mlsanity.com/paper/2112.15131/) (2021)
- [OODIn: An Optimised On-Device Inference Framework for Heterogeneous Mobile Devices](https://mlsanity.com/paper/2106.04723/) (2021)
- [SensiX++: Bringing MLOPs and Multi-tenant Model Serving to Sensory Edge Devices](https://mlsanity.com/paper/2109.03947/) (2021)
- [Turbo: Opportunistic Enhancement for Edge Video Analytics](https://mlsanity.com/paper/2207.00172/) (2022)
- [Edge Video Analytics: A Survey on Applications, Systems and Enabling Techniques](https://mlsanity.com/paper/2211.15751/) (2022)
- [Boosting DNN Cold Inference on Edge Devices](https://mlsanity.com/paper/2206.07446/) (2022)
- [Mind Your Heart: Stealthy Backdoor Attack on Dynamic Deep Neural Network in Edge Computing](https://mlsanity.com/paper/2212.11751/) (2022)
- [RED: A Systematic Real-Time Scheduling Approach for Robotic Environmental Dynamics](https://mlsanity.com/paper/2308.15368/) (2023)
- [Optimising Resource Management for Embedded Machine Learning](https://mlsanity.com/paper/2105.03608/) (2021)
- [TFormer: A Transmission-Friendly ViT Model for IoT Devices](https://mlsanity.com/paper/2302.07734/) (2023)
- [Miriam: Exploiting Elastic Kernels for Real-time Multi-DNN Inference on Edge GPU](https://mlsanity.com/paper/2307.04339/) (2023)
- [MOSEL: Inference Serving Using Dynamic Modality Selection](https://mlsanity.com/paper/2310.18481/) (2023)
- [YONO: Modeling Multiple Heterogeneous Neural Networks on Microcontrollers](https://mlsanity.com/paper/2203.03794/) (2022)
- [FrameHopper: Selective Processing of Video Frames in Detection-driven Real-Time Video Analytics](https://mlsanity.com/paper/2203.11493/) (2022)
- [SuperFedNAS: Cost-Efficient Federated Neural Architecture Search for On-Device Inference](https://mlsanity.com/paper/2301.10879/) (2023)

## Abstract (arXiv)

Mobile vision systems such as smartphones, drones, and augmented-reality headsets are revolutionizing our lives. These systems usually run multiple applications concurrently and their available resources at runtime are dynamic due to events such as starting new applications, closing existing applications, and application priority changes. In this paper, we present NestDNN, a framework that takes the dynamics of runtime resources into account to enable resource-aware multi-tenant on-device deep learning for mobile vision systems. NestDNN enables each deep learning model to offer flexible resource-accuracy trade-offs. At runtime, it dynamically selects the optimal resource-accuracy trade-off for each deep learning model to fit the model's resource demand to the system's available runtime resources. In doing so, NestDNN efficiently utilizes the limited resources in mobile vision systems to jointly maximize the performance of all the concurrently running applications. Our experiments show that compared to the resource-agnostic status quo approach, NestDNN achieves as much as 4.2% increase in inference accuracy, 2.0x increase in video frame processing rate and 1.7x reduction on energy consumption.