mlsanityData to 7 Oct 2026

On-Device Language Models: A Comprehensive Review

arXiv 2409.00088 ↗First posted 26 Aug 2024cs.CLcited by 101

Not on the map. Why?

Cited by 20 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The advent of large language models (LLMs) revolutionized natural language processing applications, and running LLMs on edge devices has become increasingly attractive for reasons including reduced latency, data localization, and personalized user experiences. This comprehensive review examines the challenges of deploying computationally expensive LLMs on resource-constrained devices and explores innovative solutions across multiple domains. The paper investigates the development of on-device language models, their efficient architectures, including parameter sharing and modular designs, as well as state-of-the-art compression techniques like quantization, pruning, and knowledge distillation. Hardware acceleration strategies and collaborative edge-cloud deployment approaches are analyzed, highlighting the intricate balance between performance and resource utilization. Case studies of on-device language models from major mobile manufacturers demonstrate real-world applications and potential benefits. The review also addresses critical aspects such as adaptive learning, multi-modal capabilities, and personalization. By identifying key research directions and open challenges, this paper provides a roadmap for future advancements in on-device language models, emphasizing the need for interdisciplinary efforts to realize the full potential of ubiquitous, intelligent computing while ensuring responsible and ethical deployment. For a comprehensive review of research work and educational resources on on-device large language models (LLMs), please visit https://github.com/NexaAI/Awesome-LLMs-on-device. To download and run on-device LLMs, visit https://www.nexaai.com/models.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Builds on 131 of its 178 references that are in the map. First posted on arXiv 26 Aug 2024.

Builds on

131 of 178 references.

References · 131

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
38 more
  1. 2023Qwen Technical Reportcited by 3,273
  2. 2023Mistral 7Bcited by 3,065
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  7. 2024Mixtral of Expertscited by 1,785
  8. 2019Root Mean Square Layer Normalizationcited by 1,425
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  11. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  13. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  14. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  15. 2024Yi: Open Foundation Models by 01.AIcited by 706
  16. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  17. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  18. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  19. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  20. 2023Textbooks Are All You Needcited by 561
  21. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  22. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  23. 2023The Falcon Series of Open Language Modelscited by 526
  24. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  25. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  26. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  27. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  28. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  29. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  30. 2024InternLM2 Technical Reportcited by 391
  31. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  32. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  33. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  34. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  35. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  36. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  37. 2023Towards Generalist Biomedical AIcited by 286
  38. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257

50 most cited of 131 All 131 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 50; 2026: 43.

101 papers.

Sort:

By area:

  1. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  2. 2024Small Language Models: Survey, Measurements, and Insightscited by 109
  3. 2024Scaling Wearable Foundation Modelscited by 57
  4. 2025Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Modelscited by 44
  5. 2024AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generationcited by 31
  6. 2025Division-of-Thoughts: Harnessing Hybrid Language Model Synergy for Efficient On-Device Agentscited by 24
  7. 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agentscited by 21
  8. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 17
  9. 2024WDMoE: Wireless Distributed Mixture of Experts for Large Language Modelscited by 12
  10. 2025Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoCcited by 11
  11. 2025Cognitive Edge Computing: A Comprehensive Survey on Optimizing Large Models and AI Agents for Pervasive Deploymentcited by 10
  12. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
38 more
  1. 2025AgentProg: Empowering Long-Horizon GUI Agents with Program-Guided Context Managementcited by 10
  2. 2024Privacy-preserved LLM Cascade via CoT-enhanced Policy Learningcited by 8
  3. 2025Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decodingcited by 8
  4. 2026Forget to Improve: On-Device LLM-Agent Continual Learning via Budget-Curated Memorycited by 8
  5. 2025SlimCaching: Edge Caching of Mixture-of-Experts for Distributed Inferencecited by 7
  6. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniquescited by 5
  7. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 5
  8. 2026CoreQ: Learning-Free Mismatch Correction and Successive Rounding for Quantizationcited by 5
  9. 2025PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computingcited by 4
  10. 2025Federated Intelligence: When Large AI Models Meet Federated Fine-Tuning and Collaborative Reasoning at the Network Edgecited by 4
  11. 2025DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Modelscited by 4
  12. 2025Bridging On-Device and Cloud LLMs for Collaborative Reasoning: A Unified Methodology for Local Routing and Post-Trainingcited by 4
  13. 2025Less LLM, More Documents: Searching for Improved RAGcited by 4
  14. 2025Disaster Management in the Era of Agentic AI Systems: A Vision for Collective Human-Machine Intelligence for Augmented Resiliencecited by 4
  15. 2026ECG-Agent: On-Device Tool-Calling Agent for ECG Multi-Turn Dialoguecited by 4
  16. 2026Rethinking KV Cache Eviction via a Unified Information-Theoretic Objectivecited by 4
  17. 2025Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inferencecited by 3
  18. 2025Generative AI for Autonomous Driving: A Reviewcited by 3
  19. 2026Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Surveycited by 3
  20. 2026ScrapMem: A Bio-inspired Framework for On-device Personalized Agent Memory via Optical Forgettingcited by 3
  21. 2024Exploring Gen-AI applications in building research and industry: A reviewcited by 2
  22. 2024Creation of AI-driven Smart Spaces for Enhanced Indoor Environments -- A Surveycited by 2
  23. 2025Advancing Generative Artificial Intelligence and Large Language Models for Demand Side Management with Internet of Electric Vehiclescited by 2
  24. 2025SmartBench: Is Your LLM Truly a Good Chinese Smartphone Assistant?cited by 2
  25. 2025On-Device LLM for Context-Aware Wi-Fi Roamingcited by 2
  26. 2025Logit Arithmetic Elicits Long Reasoning Capabilities Without Trainingcited by 2
  27. 2025Sandwich: Joint Configuration Search and Hot-Switching for Efficient CPU LLM Servingcited by 2
  28. 2025Edge Artificial Intelligence: A Systematic Review of Evolution, Taxonomic Frameworks, and Future Horizonscited by 2
  29. 2025Logit Arithmetic Elicits Long Reasoning Capabilities Without Trainingcited by 2
  30. 2025On-device System of Compositional Multi-tasking in Large Language Modelscited by 2
  31. 2026Joint Continual Learning of Local Language Models and Cloud Offloading Decisions with Budget Constraintscited by 2
  32. 2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gainscited by 2
  33. 2026DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edgecited by 2
  34. 2026Parameter-Free Encoders Remain Viable for RDB Foundation Modelscited by 2
  35. 2025Good things come in small packages: Should we build AI clusters with Lite-GPUs?cited by 1
  36. 2025P/D-Device: Disaggregated Large Language Model between Cloud and Devicescited by 1
  37. 2025Decoupling Understanding from Reasoning via Problem Space Mapping for Small-Scale Model Reasoningcited by 1
  38. 2025Cloud-Device Collaborative Agents for Sequential Recommendationcited by 1

50 most cited of 101 All 101 (JSON)