# Efficient Deep Learning Infrastructures for Embedded Computing Systems: A Comprehensive Survey and Future Envision

arXiv: https://arxiv.org/abs/2411.01431 · HTML: https://mlsanity.com/paper/2411.01431/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 81 references that are in the map.
- First posted on arXiv 3 Nov 2024.

## Builds on (most cited first, 30 of 41)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://mlsanity.com/paper/1602.01528/) (2016)
- [MCUNet: Tiny Deep Learning on IoT Devices](https://mlsanity.com/paper/2007.10319/) (2020)
- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [Training Neural Networks with Fixed Sparse Masks](https://mlsanity.com/paper/2111.09839/) (2021)
- [Self-Distillation Amplifies Regularization in Hilbert Space](https://mlsanity.com/paper/2002.05715/) (2020)
- [You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection](https://mlsanity.com/paper/2106.00666/) (2021)
- [Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better](https://mlsanity.com/paper/2106.08962/) (2021)
- [Fast Transformers with Clustered Attention](https://mlsanity.com/paper/2007.04825/) (2020)
- [BRP-NAS: Prediction-based NAS using GCNs](https://mlsanity.com/paper/2007.08668/) (2020)
- [MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning](https://mlsanity.com/paper/2110.15352/) (2021)
- [CHIP: CHannel Independence-based Pruning for Compact Neural Networks](https://mlsanity.com/paper/2110.13981/) (2021)
- [Towards Explainable Deep Neural Networks (xDNN)](https://mlsanity.com/paper/1912.02523/) (2019)
- [NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing](https://mlsanity.com/paper/2006.07116/) (2020)
- [NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks](https://mlsanity.com/paper/2110.05668/) (2021)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [Learning Best Combination for Efficient N:M Sparsity](https://mlsanity.com/paper/2206.06662/) (2022)
- [XNOR Neural Engine: a Hardware Accelerator IP for 21.6 fJ/op Binary Neural Network Inference](https://mlsanity.com/paper/1807.03010/) (2018)
- [An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers](https://mlsanity.com/paper/2208.06118/) (2022)
- [Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning](https://mlsanity.com/paper/2305.15212/) (2023)
- [SkyNet: a Hardware-Efficient Method for Object Detection and Tracking on Embedded Systems](https://mlsanity.com/paper/1909.09709/) (2019)
- [Preparing Lessons: Improve Knowledge Distillation with Better Supervision](https://mlsanity.com/paper/1911.07471/) (2019)
- [Speedy Performance Estimation for Neural Architecture Search](https://mlsanity.com/paper/2006.04492/) (2020)
- [Bringing AI To Edge: From Deep Learning's Perspective](https://mlsanity.com/paper/2011.14808/) (2020)
- [NAS-Bench-x11 and the Power of Learning Curves](https://mlsanity.com/paper/2111.03602/) (2021)
- [RED : Looking for Redundancies for Data-Free Structured Compression of Deep Neural Networks](https://mlsanity.com/paper/2105.14797/) (2021)

## Built on by (most cited first, 5 of 5)

- [LILogic Net: Compact Logic Gate Networks with Learnable Connectivity for Efficient Hardware Deployment](https://mlsanity.com/paper/2511.12340/) (2025)
- [Stochastic Layer-wise Learning: Scalable and Efficient Alternative to Backpropagation](https://mlsanity.com/paper/2505.05181/) (2025)
- [Exploring Deep-to-Shallow Transformable Neural Networks for Intelligent Embedded Systems](https://mlsanity.com/paper/2512.19731/) (2025)
- [Resource-Efficient RGB-Only Action Recognition for Edge Deployment](https://mlsanity.com/paper/2602.10818/) (2026)
- [Towards automatic smoke detector inspection: Recognition of the smoke detectors in industrial facilities and preparation for future drone integration](https://mlsanity.com/paper/2603.24850/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) have recently achieved impressive success across a wide range of real-world vision and language processing tasks, spanning from image classification to many other downstream vision tasks, such as object detection, tracking, and segmentation. However, previous well-established DNNs, despite being able to maintain superior accuracy, have also been evolving to be deeper and wider and thus inevitably necessitate prohibitive computational resources for both training and inference. This trend further enlarges the computational gap between computation-intensive DNNs and resource-constrained embedded computing systems, making it challenging to deploy powerful DNNs upon real-world embedded computing systems towards ubiquitous embedded intelligence. To alleviate the above computational gap and enable ubiquitous embedded intelligence, we, in this survey, focus on discussing recent efficient deep learning infrastructures for embedded computing systems, spanning from training to inference, from manual to automated, from convolutional neural networks to transformers, from transformers to vision transformers, from vision models to large language models, from software to hardware, and from algorithms to applications. Specifically, we discuss recent efficient deep learning infrastructures for embedded computing systems from the lens of (1) efficient manual network design for embedded computing systems, (2) efficient automated network design for embedded computing systems, (3) efficient network compression for embedded computing systems, (4) efficient on-device learning for embedded computing systems, (5) efficient large language models for embedded computing systems, (6) efficient deep learning software and hardware for embedded computing systems, and (7) efficient intelligent applications for embedded computing systems.