# Tiny Machine Learning: Progress and Futures

arXiv: https://arxiv.org/abs/2403.19076 · HTML: https://mlsanity.com/paper/2403.19076/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 103 of its 154 references that are in the map.
- First posted on arXiv 28 Mar 2024.

## Builds on (most cited first, 30 of 103)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)

## Built on by (most cited first, 30 of 40)

- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [Streaming Deep Reinforcement Learning Finally Works](https://mlsanity.com/paper/2410.14606/) (2024)
- [A Survey on Collaborative Mechanisms Between Large and Small Language Models](https://mlsanity.com/paper/2505.07460/) (2025)
- [TeleSparse: Practical Privacy-Preserving Verification of Deep Neural Networks](https://mlsanity.com/paper/2504.19274/) (2025)
- [Self-Learning for Personalized Keyword Spotting on Ultra-Low-Power Audio Sensors](https://mlsanity.com/paper/2408.12481/) (2024)
- [On-Device Training of Fully Quantized Deep Neural Networks on Cortex-M Microcontrollers](https://mlsanity.com/paper/2407.10734/) (2024)
- [Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?](https://mlsanity.com/paper/2504.09685/) (2025)
- [From Tiny Machine Learning to Tiny Deep Learning: A Survey](https://mlsanity.com/paper/2506.18927/) (2025)
- [Revisiting Matrix Sketching in Linear Bandits: Achieving Sublinear Regret via Dyadic Block Sketching](https://mlsanity.com/paper/2410.10258/) (2024)
- [From LLMs to Edge: Parameter-Efficient Fine-Tuning on Edge Devices](https://mlsanity.com/paper/2507.23536/) (2025)
- [Cycle Training with Semi-Supervised Domain Adaptation: Bridging Accuracy and Efficiency for Real-Time Mobile Scene Detection](https://mlsanity.com/paper/2504.09297/) (2025)
- [A Multicore and Edge TPU-Accelerated Multimodal TinyML System for Livestock Behavior Recognition](https://mlsanity.com/paper/2504.11467/) (2025)
- [Rounding-Guided Backdoor Injection in Deep Learning Model Quantization](https://mlsanity.com/paper/2510.09647/) (2025)
- [A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene](https://mlsanity.com/paper/2404.11249/) (2024)
- [TinyML NLP Scheme for Semantic Wireless Sentiment Classification with Privacy Preservation](https://mlsanity.com/paper/2411.06291/) (2024)
- [Edge Intelligence for Wildlife Conservation: Real-Time Hornbill Call Classification Using TinyML](https://mlsanity.com/paper/2504.12272/) (2025)
- [Hybrid Knowledge Transfer through Attention and Logit Distillation for On-Device Vision Systems in Agricultural IoT](https://mlsanity.com/paper/2504.16128/) (2025)
- [Edge Artificial Intelligence: A Systematic Review of Evolution, Taxonomic Frameworks, and Future Horizons](https://mlsanity.com/paper/2510.01439/) (2025)
- [LightCL: Compact Continual Learning with Low Memory Footprint For Edge Device](https://mlsanity.com/paper/2407.10545/) (2024)
- [Onboard Optimization and Learning: A Survey](https://mlsanity.com/paper/2505.08793/) (2025)
- [Designing Object Detection Models for TinyML: Foundations, Comparative Analysis, Challenges, and Emerging Solutions](https://mlsanity.com/paper/2508.08352/) (2025)
- [AI Compute Architecture and Evolution Trends](https://mlsanity.com/paper/2508.21394/) (2025)
- [AdaBet: Gradient-free Layer Selection for Efficient Training of Deep Neural Networks](https://mlsanity.com/paper/2510.03101/) (2025)
- [RISC-V Based TinyML Accelerator for Depthwise Separable Convolutions in Edge AI](https://mlsanity.com/paper/2511.21232/) (2025)
- [The Weight of a Bit: EMFI Sensitivity Analysis of Embedded Deep Learning Models](https://mlsanity.com/paper/2602.16309/) (2026)
- [TrainDeeploy: Hardware-Accelerated Parameter-Efficient Fine-Tuning of Small Transformer Models at the Extreme Edge](https://mlsanity.com/paper/2603.09511/) (2026)
- [EmbBERT: Attention Under 2 MB Memory](https://mlsanity.com/paper/2502.10001/) (2025)
- [Compound and Parallel Modes of Tropical Convolutional Neural Networks](https://mlsanity.com/paper/2504.06881/) (2025)
- [Integration of TinyML and LargeML: A Survey of 6G and Beyond](https://mlsanity.com/paper/2505.15854/) (2025)
- [Beyond ImageNet: Understanding Cross-Dataset Robustness of Lightweight Vision Models](https://mlsanity.com/paper/2511.00335/) (2025)

## Abstract (arXiv)

Tiny Machine Learning (TinyML) is a new frontier of machine learning. By squeezing deep learning models into billions of IoT devices and microcontrollers (MCUs), we expand the scope of AI applications and enable ubiquitous intelligence. However, TinyML is challenging due to hardware constraints: the tiny memory resource makes it difficult to hold deep learning models designed for cloud and mobile platforms. There is also limited compiler and inference engine support for bare-metal devices. Therefore, we need to co-design the algorithm and system stack to enable TinyML. In this review, we will first discuss the definition, challenges, and applications of TinyML. We then survey the recent progress in TinyML and deep learning on MCUs. Next, we will introduce MCUNet, showing how we can achieve ImageNet-scale AI applications on IoT devices with system-algorithm co-design. We will further extend the solution from inference to training and introduce tiny on-device training techniques. Finally, we present future directions in this area. Today's large model might be tomorrow's tiny model. The scope of TinyML should evolve and adapt over time.