# LegoDNN: Block-grained Scaling of Deep Neural Networks for Mobile Vision

arXiv: https://arxiv.org/abs/2112.09852 · HTML: https://mlsanity.com/paper/2112.09852/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 68 references that are in the map.
- First posted on arXiv 18 Dec 2021.

## Builds on (most cited first, 30 of 39)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [Residual Attention Network for Image Classification](https://mlsanity.com/paper/1704.06904/) (2017)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [HRank: Filter Pruning using High-Rank Feature Map](https://mlsanity.com/paper/2002.10179/) (2020)
- [A Survey of the Recent Architectures of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1901.06032/) (2019)
- [BlockDrop: Dynamic Inference Paths in Residual Networks](https://mlsanity.com/paper/1711.08393/) (2017)
- [Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications](https://mlsanity.com/paper/1511.06530/) (2015)
- [Convolutional Networks with Adaptive Inference Graphs](https://mlsanity.com/paper/1711.11503/) (2017)
- [Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning](https://mlsanity.com/paper/1611.05128/) (2016)
- [Distributed Deep Neural Networks over the Cloud, the Edge and End Devices](https://mlsanity.com/paper/1709.01921/) (2017)
- [Provable Filter Pruning for Efficient Neural Networks](https://mlsanity.com/paper/1911.07412/) (2019)
- [Channel Gating Neural Networks](https://mlsanity.com/paper/1805.12549/) (2018)
- [And the Bit Goes Down: Revisiting the Quantization of Neural Networks](https://mlsanity.com/paper/1907.05686/) (2019)

## Built on by (most cited first, 20 of 20)

- [LLMCad: Fast and Scalable On-device Large Language Model Inference](https://mlsanity.com/paper/2309.04255/) (2023)
- [SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget](https://mlsanity.com/paper/2308.15030/) (2023)
- [AdaptiveNet: Post-deployment Neural Architecture Adaptation for Diverse Edge Environments](https://mlsanity.com/paper/2303.07129/) (2023)
- [MergeSFL: Split Federated Learning with Feature Merging and Batch Size Regulation](https://mlsanity.com/paper/2311.13348/) (2023)
- [Edge Video Analytics: A Survey on Applications, Systems and Enabling Techniques](https://mlsanity.com/paper/2211.15751/) (2022)
- [Elastic On-Device LLM Service](https://mlsanity.com/paper/2409.09071/) (2024)
- [Panopticus: Omnidirectional 3D Object Detection on Resource-constrained Edge Devices](https://mlsanity.com/paper/2410.01270/) (2024)
- [ECLM: Efficient Edge-Cloud Collaborative Learning with Continuous Environment Adaptation](https://mlsanity.com/paper/2311.11083/) (2023)
- [CARIn: Constraint-Aware and Responsive Inference on Heterogeneous Devices for Single- and Multi-DNN Workloads](https://mlsanity.com/paper/2409.01089/) (2024)
- [DARDA: Domain-Aware Real-Time Dynamic Neural Network Adaptation](https://mlsanity.com/paper/2409.09753/) (2024)
- [AdaScale: Dynamic Context-aware DNN Scaling via Automated Adaptation Loop on Mobile Devices](https://mlsanity.com/paper/2412.00724/) (2024)
- [Hierarchical Memory Pool Based Edge Semi-Supervised Continual Learning Method](https://mlsanity.com/paper/2303.11952/) (2023)
- [Generative Model for Models: Rapid DNN Customization for Diverse Tasks and Resource Constraints](https://mlsanity.com/paper/2308.15003/) (2023)
- [BRIEDGE: EEG-Adaptive Edge AI for Multi-Brain to Multi-Robot Interaction](https://mlsanity.com/paper/2403.15432/) (2024)
- [Moss: Proxy Model-based Full-Weight Aggregation in Federated Learning with Heterogeneous Models](https://mlsanity.com/paper/2503.10218/) (2025)
- [Towards Adaptive Deep Learning: Model Elasticity via Prune-and-Grow CNN Architectures](https://mlsanity.com/paper/2505.11569/) (2025)
- [XTransfer: Modality-Agnostic Few-Shot Model Transfer for Human Sensing at the Edge](https://mlsanity.com/paper/2506.22726/) (2025)
- [AutoTailor: Automatic and Efficient Adaptive Model Deployment for Diverse Edge Devices](https://mlsanity.com/paper/2511.22355/) (2025)
- [MO-SAE:Multi-Objective Stacked Autoencoders Optimization for Edge Anomaly Detection](https://mlsanity.com/paper/2603.13895/) (2026)
- [Elastoformer: Enabling Dynamic Adaptivity via Elastic Model Transformation](https://mlsanity.com/paper/2609.10018/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) have become ubiquitous techniques in mobile and embedded systems for applications such as image/object recognition and classification. The trend of executing multiple DNNs simultaneously exacerbate the existing limitations of meeting stringent latency/accuracy requirements on resource constrained mobile devices. The prior art sheds light on exploring the accuracy-resource tradeoff by scaling the model sizes in accordance to resource dynamics. However, such model scaling approaches face to imminent challenges: (i) large space exploration of model sizes, and (ii) prohibitively long training time for different model combinations. In this paper, we present LegoDNN, a lightweight, block-grained scaling solution for running multi-DNN workloads in mobile vision systems. LegoDNN guarantees short model training times by only extracting and training a small number of common blocks (e.g. 5 in VGG and 8 in ResNet) in a DNN. At run-time, LegoDNN optimally combines the descendant models of these blocks to maximize accuracy under specific resources and latency constraints, while reducing switching overhead via smart block-level scaling of the DNN. We implement LegoDNN in TensorFlow Lite and extensively evaluate it against state-of-the-art techniques (FLOP scaling, knowledge distillation and model compression) using a set of 12 popular DNN models. Evaluation results show that LegoDNN provides 1,296x to 279,936x more options in model sizes without increasing training time, thus achieving as much as 31.74% improvement in inference accuracy and 71.07% reduction in scaling energy consumptions.