mlsanityData to 7 Oct 2026

Lightweight Deep Learning for Resource-Constrained Environments: A Survey

arXiv 2404.07236 ↗First posted 8 Apr 2024cs.CV cs.LGcited by 38

Not on the map. Why?

Abstract (arXiv)

Over the past decade, the dominance of deep learning has prevailed across various domains of artificial intelligence, including natural language processing, computer vision, and biomedical signal processing. While there have been remarkable improvements in model accuracy, deploying these models on lightweight devices, such as mobile phones and microcontrollers, is constrained by limited resources. In this survey, we provide comprehensive design guidance tailored for these devices, detailing the meticulous design of lightweight models, compression methods, and hardware acceleration strategies. The principal goal of this work is to explore methods and concepts for getting around hardware constraints without compromising the model's accuracy. Additionally, we explore two notable paths for lightweight deep learning in the future: deployment techniques for TinyML and Large Language Models. Although these paths undoubtedly have potential, they also present significant challenges, encouraging research into unexplored areas.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 149 of its 240 references that are in the map. First posted on arXiv 8 Apr 2024.

Builds on

149 of 240 references.

References · 149

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  11. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  12. 2016Densely Connected Convolutional Networkscited by 6,691
38 more
  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
  8. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  10. 2019Searching for MobileNetV3cited by 1,984
  11. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  12. 2020Reformer: The Efficient Transformercited by 1,658
  13. 2020Rethinking Attention with Performerscited by 1,584
  14. 2020Knowledge Distillation: A Surveycited by 1,578
  15. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  16. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  17. 2016Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learningcited by 1,509
  18. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  19. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,306
  20. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  21. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  22. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  23. 2017Mixed Precision Trainingcited by 1,135
  24. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  25. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  26. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,127
  27. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  28. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  29. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  30. 2020Designing Network Design Spacescited by 954
  31. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  32. 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
  33. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  34. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  35. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  36. 2020Efficient Transformers: A Surveycited by 784
  37. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  38. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697

50 most cited of 149 All 149 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 24; 2026: 9.

38 papers.

Sort:

By area:

  1. 2024Light-T2M: A Lightweight and Fast Model for Text-to-motion Generationcited by 21
  2. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipelinecited by 20
  3. 2025Foundation Models for Time Series: A Surveycited by 19
  4. 2025Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applicationscited by 6
  5. 2024Optimizing RAG Techniques for Automotive Industry PDF Chatbots: A Case Study with Locally Deployed Ollama Modelscited by 4
  6. 2025LGPS: A Lightweight GAN-Based Approach for Polyp Segmentation in Colonoscopy Imagescited by 3
  7. 2025Graph Learning-Driven Multi-Vessel Association: Fusing Multimodal Data for Maritime Intelligencecited by 3
  8. 2025Membership and Memorization in LLM Knowledge Distillationcited by 3
  9. 2024Unlocking Real-Time Fluorescence Lifetime Imaging: Multi-Pixel Parallelism for FPGA-Accelerated Processingcited by 2
  10. 2025Pruning-Based TinyML Optimization of Machine Learning Models for Anomaly Detection in Electric Vehicle Charging Infrastructurecited by 2
  11. 2025Single-shot Star-convex Polygon-based Instance Segmentation for Spatially-correlated Biomedical Objectscited by 2
  12. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
26 more
  1. 2024Deep Learning Based XIoT Malware Analysis: A Comprehensive Survey, Taxonomy, and Research Challengescited by 1
  2. 2025RadarNeXt: Real-Time and Reliable 3D Object Detector Based On 4D mmWave Imaging Radarcited by 1
  3. 2025Swapped Logit Distillation via Bi-level Teacher Alignmentcited by 1
  4. 2025SVarM: Linear Support Varifold Machines for Classification and Regression on Geometric Datacited by 1
  5. 2025MSPT: A Lightweight Face Image Quality Assessment Method with Multi-stage Progressive Trainingcited by 1
  6. 2025Designing Object Detection Models for TinyML: Foundations, Comparative Analysis, Challenges, and Emerging Solutionscited by 1
  7. 2025Decentor-V: Lightweight ML Training on Low-Power RISC-V Edge Devicescited by 1
  8. 2026CarbonEdge: Carbon-Aware Deep Learning Inference Framework for Sustainable Edge Computingcited by 1
  9. 2026Equivariant Multi-agent Reinforcement Learning for Multimodal Vehicle-to-Infrastructure Systemscited by 1
  10. 2024InFiConD: Interactive No-code Fine-tuning with Concept-based Knowledge Distillationcited by 0
  11. 2025Lightweight and Interpretable Transformer via Mixed Graph Algorithm Unrolling for Traffic Forecastcited by 0
  12. 2025Smooth Model Compression without Fine-Tuningcited by 0
  13. 2025Optimizing Split Learning Latency in TinyML-Based IoT Systemscited by 0
  14. 2025Machine Learning in Micromobility: A Systematic Review of Datasets, Techniques, and Applicationscited by 0
  15. 2025M3ET: Efficient Vision-Language Learning for Robotics based on Multimodal Mamba-Enhanced Transformercited by 0
  16. 2025FERD: Fairness-Enhanced Data-Free Robustness Distillationcited by 0
  17. 2025Beyond ImageNet: Understanding Cross-Dataset Robustness of Lightweight Vision Modelscited by 0
  18. 2025ECCO: Leveraging Cross-Camera Correlations for Efficient Live Video Continuous Learningcited by 0
  19. 2025A Lightweight Multi-Scale Attention Framework for Real-Time Spinal Endoscopic Instance Segmentationcited by 0
  20. 2026U-FedTomAtt: Ultra-lightweight Federated Learning with Attention for Tomato Disease Recognitioncited by 0
  21. 2026Low-Rank Compression of Pretrained Models via Randomized Subspace Iterationcited by 0
  22. 2026TinyNeRV: Compact Neural Video Representations via Capacity Scaling, Distillation, and Low-Precision Inferencecited by 0
  23. 2026MARR: Module-Adaptive Residual Reconstruction for Low-Bit Post-Training Quantizationcited by 0
  24. 2026Tiny Collaborative Inference for Occlusion-Robust Object Detectioncited by 0
  25. 2026UtVAA: Ultra-tiny Vision Transformer with Affix Attention for Mobile Image Classificationcited by 0
  26. 2026HiRo: A Compact Four-Directional Hierarchical Reservoir Token-Mixer for Efficient Image Classificationcited by 0