mlsanityData to 7 Oct 2026

Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better

arXiv 2106.08962 ↗First posted 16 Jun 2021cs.LGcited by 153

Not on the map. Why?

Abstract (arXiv)

Deep Learning has revolutionized the fields of computer vision, natural language understanding, speech recognition, information retrieval and more. However, with the progressive improvements in deep learning models, their number of parameters, latency, resources required to train, etc. have all have increased significantly. Consequently, it has become important to pay attention to these footprint metrics of a model as well, not just its quality. We present and motivate the problem of efficiency in deep learning, followed by a thorough survey of the five core areas of model efficiency (spanning modeling techniques, infrastructure, and hardware) and the seminal work there. We also present an experiment-based guide along with code, for practitioners to optimize their model training and deployment. We believe this is the first comprehensive survey in the efficient deep learning space that covers the landscape of model efficiency from modeling techniques to hardware support. Our hope is that this survey would provide the reader with the mental model and the necessary understanding of the field to apply generic efficiency techniques to immediately get significant improvements, and also equip them with ideas for further research and experimentation to achieve additional gains.

Key facts. Cited by 153 papers in the map (data through 7 Oct 2026). Builds on 83 of its 181 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

83 of 181 references.

References · 83

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  11. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  12. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
38 more
  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  2. 2014Going Deeper with Convolutionscited by 4,231
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  5. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  10. 2019Searching for MobileNetV3cited by 1,984
  11. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  12. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  13. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  14. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  15. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  16. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  17. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  18. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  19. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  20. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  21. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  22. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  23. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  24. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  25. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  26. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  27. 2020Efficient Transformers: A Surveycited by 784
  28. 2016Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimizationcited by 688
  29. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  30. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  31. 2018Rethinking the Value of Network Pruningcited by 595
  32. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  33. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  34. 2017Progressive Neural Architecture Searchcited by 483
  35. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  36. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  37. 2019The State of Sparsity in Deep Neural Networkscited by 471
  38. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 427

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 11; 2023: 29; 2024: 46; 2025: 34; 2026: 27.

153 papers.

Sort:

By area:

  1. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  2. 2021Implicit Neural Representations for Image Compressioncited by 161
  3. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  4. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  5. 2023Structured Pruning for Deep Convolutional Neural Networks: A surveycited by 122
  6. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  7. 2021The Efficiency Misnomercited by 98
  8. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  9. 2023On the Impact of Calibration Data in Post-training Quantization and Pruningcited by 51
  10. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  11. 2022A Survey of Active Learning for Natural Language Processingcited by 40
  12. 2024MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMscited by 39
38 more
  1. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  2. 2024Efficient Learning With Sine-Activated Low-rank Matricescited by 22
  3. 2023Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiencescited by 21
  4. 2022pNLP-Mixer: an Efficient all-MLP Architecture for Languagecited by 19
  5. 2023Towards Trustworthy Dataset Distillationcited by 14
  6. 2022Differentially Private Model Compressioncited by 13
  7. 2023ResMem: Learn what you can and memorize the restcited by 13
  8. 2023Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecyclecited by 13
  9. 2021Understanding Performance Problems in Deep Learning Systemscited by 12
  10. 2024When IoT Meet LLMs: Applications and Challengescited by 11
  11. 2026ELT: Elastic Looped Transformers for Visual Generationcited by 11
  12. 2021A Lightweight and Accurate Spatial-Temporal Transformer for Traffic Forecastingcited by 9
  13. 2022What Do Compressed Multilingual Machine Translation Models Forget?cited by 9
  14. 2023PELA: Learning Parameter-Efficient Models with Low-Rank Approximationcited by 9
  15. 2023Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Reviewcited by 9
  16. 2024HGQ: High Granularity Quantization for Real-time Neural Networks on FPGAscited by 9
  17. 2024WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Averagecited by 9
  18. 2025ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Datasetcited by 9
  19. 2023Global Proxy-based Hard Mining for Visual Place Recognitioncited by 8
  20. 2024Brain-inspired Artificial Intelligence: A Comprehensive Reviewcited by 8
  21. 2025Task-Specific Generative Dataset Distillation with Difficulty-Guided Samplingcited by 8
  22. 2025Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Frameworkcited by 8
  23. 2025Generative Artificial Intelligence in Medical Imaging: Foundations, Progress, and Clinical Translationcited by 8
  24. 2023Model Pruning Enables Localized and Efficient Federated Learning for Yield Forecasting and Data Sharingcited by 7
  25. 2023Sparse-DySta: Sparsity-Aware Dynamic and Static Scheduling for Sparse Multi-DNN Workloadscited by 6
  26. 2023Prototypical Self-Explainable Models Without Re-trainingcited by 6
  27. 2024Talaria: Interactively Optimizing Machine Learning Models for Efficient Inferencecited by 6
  28. 2024Imaging foundation model for universal enhancement of non-ideal measurement CTcited by 6
  29. 2024LAuReL: Learned Augmented Residual Layercited by 6
  30. 2025Decoupled Audio-Visual Dataset Distillationcited by 6
  31. 2023Risk-Controlling Model Selection via Guided Bayesian Optimizationcited by 5
  32. 2024Kaleidoscope: Learnable Masks for Heterogeneous Multi-agent Reinforcement Learningcited by 5
  33. 2024Efficient Deep Learning Infrastructures for Embedded Computing Systems: A Comprehensive Survey and Future Envisioncited by 5
  34. 2025On Accelerating Edge AI: Optimizing Resource-Constrained Environmentscited by 5
  35. 2025From Tiny Machine Learning to Tiny Deep Learning: A Surveycited by 5
  36. 2022ALGAN: Anomaly Detection by Generating Pseudo Anomalous Data via Latent Variablescited by 4
  37. 2022Generalization In Multi-Objective Machine Learningcited by 4
  38. 2023Real-Time Semantic Segmentation: A Brief Survey & Comparative Study in Remote Sensingcited by 4

50 most cited of 153 All 153 (JSON)