# Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models

arXiv: https://arxiv.org/abs/2503.06027 · HTML: https://mlsanity.com/paper/2503.06027/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 99 of its 286 references that are in the map.
- First posted on arXiv 8 Mar 2025.

## Builds on (most cited first, 30 of 99)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer](https://mlsanity.com/paper/2110.02178/) (2021)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [HRank: Filter Pruning using High-Rank Feature Map](https://mlsanity.com/paper/2002.10179/) (2020)
- [DynaBERT: Dynamic BERT with Adaptive Width and Depth](https://mlsanity.com/paper/2004.04037/) (2020)
- [Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing](https://mlsanity.com/paper/1905.10083/) (2019)
- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)
- [A Fast Post-Training Pruning Framework for Transformers](https://mlsanity.com/paper/2204.09656/) (2022)
- [Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing](https://mlsanity.com/paper/1910.05316/) (2019)
- [Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspective](https://mlsanity.com/paper/2112.06409/) (2021)
- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)

## Built on by (most cited first, 30 of 46)

- [From Tiny Machine Learning to Tiny Deep Learning: A Survey](https://mlsanity.com/paper/2506.18927/) (2025)
- [Automating Versatile Time-Series Analysis with Tiny Transformers on Embedded FPGAs](https://mlsanity.com/paper/2505.17662/) (2025)
- [ECVL-ROUTER: Scenario-Aware Routing for Vision-Language Models](https://mlsanity.com/paper/2510.27256/) (2025)
- [Performance Analysis of Edge and In-Sensor AI Processors: A Comparative Review](https://mlsanity.com/paper/2603.08725/) (2026)
- [AI for Regulatory Affairs: Balancing Accuracy, Interpretability, and Computational Cost in Medical Device Classification](https://mlsanity.com/paper/2505.18695/) (2025)
- [Embodied AI with Foundation Models for Mobile Service Robots: A Systematic Review](https://mlsanity.com/paper/2505.20503/) (2025)
- [Future of Edge AI in biodiversity monitoring](https://mlsanity.com/paper/2602.13496/) (2026)
- [Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation](https://mlsanity.com/paper/2604.19444/) (2026)
- [ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference](https://mlsanity.com/paper/2605.27081/) (2026)
- [EdgeWisePersona: A Dataset for On-Device User Profiling from Natural Language Interactions](https://mlsanity.com/paper/2505.11417/) (2025)
- [Edge Artificial Intelligence: A Systematic Review of Evolution, Taxonomic Frameworks, and Future Horizons](https://mlsanity.com/paper/2510.01439/) (2025)
- [MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representation](https://mlsanity.com/paper/2602.07082/) (2026)
- [Mobile-MMLU: A Mobile Intelligence Language Understanding Benchmark](https://mlsanity.com/paper/2503.20786/) (2025)
- [Comparative Evaluation of ChatGPT and DeepSeek Across Key NLP Tasks: Strengths, Weaknesses, and Domain-Specific Performance](https://mlsanity.com/paper/2506.18501/) (2025)
- [ECR: Manifold-Guided Semantic Cues for Compact Language Models](https://mlsanity.com/paper/2601.00543/) (2026)
- [GLIMPSE : Real-Time Text Recognition and Contextual Understanding for VQA in Wearables](https://mlsanity.com/paper/2602.13479/) (2026)
- [The Internet of Physical AI Agents: Interoperability, Longevity, and the Cost of Getting It Wrong](https://mlsanity.com/paper/2603.15900/) (2026)
- [WISV: Wireless-Informed Semantic Verification for Distributed Speculative Decoding in Device-Edge LLM Inference](https://mlsanity.com/paper/2604.17701/) (2026)
- [When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference](https://mlsanity.com/paper/2605.27435/) (2026)
- [Distilling Empathy from Large Language Models](https://mlsanity.com/paper/2507.08151/) (2025)
- [Hot-Swap MarkBoard: An Efficient Black-box Watermarking Approach for Large-scale Model Distribution](https://mlsanity.com/paper/2507.20650/) (2025)
- [CSRAP: Enhanced Canvas Attention Scheduling for Real-Time Mission Critical Perception](https://mlsanity.com/paper/2508.04976/) (2025)
- [Beyond ImageNet: Understanding Cross-Dataset Robustness of Lightweight Vision Models](https://mlsanity.com/paper/2511.00335/) (2025)
- [FiCABU: A Fisher-Based, Context-Adaptive Machine Unlearning Processor for Edge AI](https://mlsanity.com/paper/2511.05605/) (2025)
- [AutoTailor: Automatic and Efficient Adaptive Model Deployment for Diverse Edge Devices](https://mlsanity.com/paper/2511.22355/) (2025)
- [Dora: QoE-Aware Hybrid Parallelism for Distributed Edge AI](https://mlsanity.com/paper/2512.10990/) (2025)
- [MD-AirComp+: Adaptive Quantization for Blind Massive Digital Over-the-Air Computation](https://mlsanity.com/paper/2602.18332/) (2026)
- [Storing Less, Finding More: How Novelty Filtering Improves Cross-Modal Retrieval on Edge Cameras](https://mlsanity.com/paper/2603.29631/) (2026)
- [MLFCIL: A Multi-Level Forgetting Mitigation Framework for Federated Class-Incremental Learning in LEO Satellites](https://mlsanity.com/paper/2604.02356/) (2026)
- [On Emotion-Sensitive Decision Making of Small Language Model Agents](https://mlsanity.com/paper/2604.06562/) (2026)

## Abstract (arXiv)

The rapid advancement of artificial intelligence (AI) technologies has led to an increasing deployment of AI models on edge and terminal devices, driven by the proliferation of the Internet of Things (IoT) and the need for real-time data processing. This survey comprehensively explores the current state, technical challenges, and future trends of on-device AI models. We define on-device AI models as those designed to perform local data processing and inference, emphasizing their characteristics such as real-time performance, resource constraints, and enhanced data privacy. The survey is structured around key themes, including the fundamental concepts of AI models, application scenarios across various domains, and the technical challenges faced in edge environments. We also discuss optimization and implementation strategies, such as data preprocessing, model compression, and hardware acceleration, which are essential for effective deployment. Furthermore, we examine the impact of emerging technologies, including edge computing and foundation models, on the evolution of on-device AI models. By providing a structured overview of the challenges, solutions, and future directions, this survey aims to facilitate further research and application of on-device AI, ultimately contributing to the advancement of intelligent systems in everyday life.