mlsanityData to 14 Oct 2026

Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques

arXiv 2112.15131 ↗First posted 30 Dec 2021cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Deep learning is pervasive in our daily life, including self-driving cars, virtual assistants, social network services, healthcare services, face recognition, etc. However, deep neural networks demand substantial compute resources during training and inference. The machine learning community has mainly focused on model-level optimizations such as architectural compression of deep learning models, while the system community has focused on implementation-level optimization. In between, various arithmetic-level optimization techniques have been proposed in the arithmetic community. This article provides a survey on resource-efficient deep learning techniques in terms of model-, arithmetic-, and implementation-level techniques and identifies the research gaps for resource-efficient deep learning techniques across the three different level techniques. Our survey clarifies the influence from higher to lower-level techniques based on our resource-efficiency metric definition and discusses the future trend for resource-efficient deep learning research.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 103 of its 211 references that are in the map. First posted on arXiv 30 Dec 2021.

Builds on

103 of 211 references.

References · 103

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  4. 2016Densely Connected Convolutional Networkscited by 6,720
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  8. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  9. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  11. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  12. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
38 more
  1. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  2. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  3. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,630
  4. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  5. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  6. 2017Mixed Precision Trainingcited by 1,156
  7. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  8. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  9. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  10. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,020
  11. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 943
  12. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  13. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  14. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  15. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  16. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  17. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  18. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  19. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  20. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 485
  21. 2013Do Deep Nets Really Need to be Deep?cited by 474
  22. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  23. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 434
  24. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  25. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 378
  26. 2015Deep Learning with Limited Numerical Precisioncited by 376
  27. 2020SplitFed: When Federated Learning Meets Split Learningcited by 310
  28. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  29. 2018Slimmable Neural Networkscited by 305
  30. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 301
  31. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 295
  32. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  33. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  34. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  35. 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 221
  36. 2016Trained Ternary Quantizationcited by 208
  37. 2016Ternary Weight Networkscited by 204
  38. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188

50 most cited of 103 All 103 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 2.

9 papers.

  1. 2023Domain Generalization in Computational Pathology: Survey and Guidelinescited by 29
  2. 2023Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecyclecited by 14
  3. 2023Real-Time Semantic Segmentation: A Brief Survey & Comparative Study in Remote Sensingcited by 4
  4. 2023Compressing the Backward Pass of Large-Scale Neural Architectures by Structured Activation Pruningcited by 3
  5. 2023PDPU: An Open-Source Posit Dot-Product Unit for Deep Learning Applicationscited by 2
  6. 2023Quantized Distillation: Optimizing Driver Activity Recognition Models for Resource-Constrained Environmentscited by 1
  7. 2023Mobiprox: Supporting Dynamic Approximate Computing on Mobilescited by 0
  8. 2024Mitigating scalability challenges in LUT-based neural networks via pruning optimisationscited by 0
  9. 2024A resource-efficient model for deep kernel learningcited by 0