mlsanityData to 7 Oct 2026

Compacting Deep Neural Networks for Internet of Things: Methods and Applications

arXiv 2103.11083 ↗First posted 20 Mar 2021cs.LG cs.AI cs.NIcited by 6

Not on the map. Why?

Abstract (arXiv)

Deep Neural Networks (DNNs) have shown great success in completing complex tasks. However, DNNs inevitably bring high computational cost and storage consumption due to the complexity of hierarchical structures, thereby hindering their wide deployment in Internet-of-Things (IoT) devices, which have limited computational capability and storage capacity. Therefore, it is a necessity to investigate the technologies to compact DNNs. Despite tremendous advances in compacting DNNs, few surveys summarize compacting-DNNs technologies, especially for IoT applications. Hence, this paper presents a comprehensive study on compacting-DNNs technologies. We categorize compacting-DNNs technologies into three major types: 1) network model compression, 2) Knowledge Distillation (KD), 3) modification of network structures. We also elaborate on the diversity of these approaches and make side-by-side comparisons. Moreover, we discuss the applications of compacted DNNs in various IoT applications and outline future directions.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 110 of its 215 references that are in the map. First posted on arXiv 20 Mar 2021.

Builds on

110 of 215 references.

References · 110

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2017Squeeze-and-Excitation Networkscited by 4,763
  7. 2014Going Deeper with Convolutionscited by 4,231
  8. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  9. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  10. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  11. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
38 more
  1. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  2. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  3. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  4. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  5. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  6. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  7. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  8. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  9. 2019Relational Knowledge Distillationcited by 873
  10. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  11. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  12. 2014Deep Learning in Neural Networks: An Overviewcited by 765
  13. 2019Contrastive Representation Distillationcited by 734
  14. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  15. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  16. 2018Born Again Neural Networkscited by 539
  17. 2019Similarity-Preserving Knowledge Distillationcited by 507
  18. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  19. 2017Progressive Neural Architecture Searchcited by 483
  20. 2013Do Deep Nets Really Need to be Deep?cited by 466
  21. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 427
  22. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  23. 2019Single Path One-Shot Neural Architecture Search with Uniform Samplingcited by 411
  24. 2019GhostNet: More Features from Cheap Operationscited by 410
  25. 2015Deep Learning with Limited Numerical Precisioncited by 368
  26. 2019Variational Information Distillation for Knowledge Transfercited by 321
  27. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  28. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  29. 2019Correlation Congruence for Knowledge Distillationcited by 280
  30. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 235
  31. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  32. 2019Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computingcited by 225
  33. 2013Predicting Parameters in Deep Learningcited by 223
  34. 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 220
  35. 2015Unifying distillation and privileged informationcited by 218
  36. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 212
  37. 2016Ternary Weight Networkscited by 195
  38. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189

50 most cited of 110 All 110 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2; 2024: 1; 2025: 1.

6 papers.

  1. 2023Enabling Resource-efficient AIoT System with Cross-level Optimization: A surveycited by 8
  2. 2022Deep Learning based Computer Vision Methods for Complex Traffic Environments Perception: A Reviewcited by 7
  3. 2022Principled Pruning of Bayesian Neural Networks through Variational Free Energy Minimizationcited by 6
  4. 2023Automated Semiconductor Defect Inspection in Scanning Electron Microscope Images: a Systematic Reviewcited by 2
  5. 2024CrowdTransfer: Enabling Crowd Knowledge Transfer in AIoT Communitycited by 0
  6. 2025TinyEcoWeedNet: Edge Efficient Real-Time Aerial Agricultural Weed Detectioncited by 0