mlsanityData to 14 Oct 2026

Fast On-device LLM Inference with NPUs

arXiv 2407.05858 ↗First posted 8 Jul 2024cs.AIcited by 62

Not on the map. Why?

Abstract (arXiv)

On-device inference for Large Language Models (LLMs), driven by increasing privacy concerns and advancements of mobile-sized models, has gained significant interest. However, even mobile-sized LLMs (e.g., Gemma-2B) encounter unacceptably high inference latency, often bottlenecked by the prefill stage in tasks like screen UI understanding. We present llm.npu, the first LLM inference system utilizing on-device Neural Processing Unit (NPU) offloading to reduce prefill latency. llm.npu enhances NPU offloading efficiency by re-constructing the prompt and model in three levels: (1) At prompt level, it divides variable-length prompts into multiple fixed-sized chunks while maintaining data dependencies; (2) At tensor level, it identifies and extracts significant outliers to run on the CPU/GPU in parallel with minimal overhead; (3) At block level, it schedules Transformer blocks in an out-of-order manner to the CPU/GPU and NPU based on their hardware affinity and sensitivity to accuracy. Compared to competitive baselines, llm.npu achieves 22.4x faster prefill speed and 30.7× energy savings on average, and up to 32.8x speedup in an end-to-end real-world application. For the first time, llm.npu achieves more than 1,000 tokens/sec prefilling for a billion-sized model.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 45 of its 97 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

45 of 97 references.

References · 45

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  3. 2023Qwen Technical Reportcited by 3,344
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  11. 2022Token Merging: Your ViT But Fastercited by 1,021
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
33 more
  1. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  2. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  5. 2022Efficiently Scaling Transformer Inferencecited by 539
  6. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  7. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  8. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  9. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  10. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  11. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 184
  12. 2023REST: Retrieval-Based Speculative Decodingcited by 150
  13. 2021Learned Token Pruning for Transformerscited by 139
  14. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  15. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  16. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  17. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  18. 2020PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruningcited by 76
  19. 2020MNN: A Universal and Efficient Inference Enginecited by 65
  20. 2017DeepCache: Principled Cache for Mobile Deep Visioncited by 57
  21. 2018A First Look at Deep Learning Apps on Smartphonescited by 47
  22. 2023Faithful Persona-based Conversational Dataset Generation with Large Language Modelscited by 47
  23. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45
  24. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  25. 2020Energy-Efficient Resource Management for Federated Edge Learning with CPU-GPU Heterogeneous Computingcited by 39
  26. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  27. 2022Mandheling: Mixed-Precision On-Device DNN Training with DSP Offloadingcited by 29
  28. 2019MobiSR: Efficient On-Device Super-Resolution through Heterogeneous Mobile Processorscited by 27
  29. 2021SpeechMoE2: Mixture-of-Experts Model with Improved Routingcited by 26
  30. 2023Mobile Foundation Model as Firmwarecited by 25
  31. 2024SmartMem: Layout Transformation Elimination and Adaptation for Efficient DNN Execution on Mobilecited by 12
  32. 2014DSP.Ear: Leveraging Co-Processor Support for Continuous Audio Sensing on Smartphonescited by 7
  33. 2024SoD2: Statically Optimizing Dynamic Deep Neural Networkcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 25; 2026: 32.

62 papers.

Sort:

By area:

  1. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  2. 2025AutoNeural: Co-Designing Vision-Language Models for NPU Inferencecited by 17
  3. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  4. 2025Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoCcited by 11
  5. 2024Understanding Large Language Models in Your Pockets: Performance Study on COTS Mobile Devicescited by 10
  6. 2024INT-FlashAttention: Enabling Flash Attention for INT8 Quantizationcited by 9
  7. 2024PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-trainingcited by 7
  8. 2025Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Executioncited by 5
  9. 2025Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computingcited by 5
  10. 2025Are We There Yet? A Measurement Study of Efficiency for LLM Applications on Mobile Devicescited by 4
  11. 2025ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inferencecited by 4
  12. 2025DynaKV: Enabling Accurate and Efficient Long-Sequence LLM Decoding on Smartphonescited by 4
38 more
  1. 2024Recall: Empowering Multimodal Embedding for Edge Devicescited by 3
  2. 2025Scaling On-Device GPU Inference for Large Generative Modelscited by 3
  3. 2025MNN-AECS: Energy Optimization for LLM Decoding on Mobile Devices via Adaptive Core Selectioncited by 3
  4. 2025CHIME: Chiplet-based Heterogeneous Near-Memory Acceleration for Edge Multimodal LLM Inferencecited by 3
  5. 2026QUOKA: Query-Oriented KV Selection For Efficient LLM Prefillcited by 3
  6. 2026The Cognitive Firewall:Securing Browser Based AI Agents Against Indirect Prompt Injection Via Hybrid Edge Cloud Defensecited by 3
  7. 2025High-Throughput LLM inference on Heterogeneous Clusterscited by 2
  8. 2025EdgeWisePersona: A Dataset for On-Device User Profiling from Natural Language Interactionscited by 2
  9. 2025P/D-Device: Disaggregated Large Language Model between Cloud and Devicescited by 2
  10. 2025Rearchitecting Datacenter Lifecycle for AI: A TCO-Driven Frameworkcited by 2
  11. 2025Edge Artificial Intelligence: A Systematic Review of Evolution, Taxonomic Frameworks, and Future Horizonscited by 2
  12. 2026Mapping Gemma3 onto an Edge Dataflow Architecturecited by 2
  13. 2026EnerInfer: Energy-Aware On-Device LLM Inferencecited by 2
  14. 2026Parameter-Free Encoders Remain Viable for RDB Foundation Modelscited by 2
  15. 2025Good things come in small packages: Should we build AI clusters with Lite-GPUs?cited by 1
  16. 2025Onboard Optimization and Learning: A Surveycited by 1
  17. 2025MobiEdit: Resource-efficient Knowledge Editing for Personalized On-device LLMscited by 1
  18. 2025Tiny but Mighty: A Software-Hardware Co-Design Approach for Efficient Multimodal Inference on Battery-Powered Small Devicescited by 1
  19. 2025PerCache: Predictive Hierarchical Cache for RAG Applications on Mobile Devicescited by 1
  20. 2026Fast Forward: Accelerating LLM Prefill with Predictive FFN Sparsitycited by 1
  21. 2026FlashMem: Supporting Modern DNN Workloads on Mobile with GPU Memory Hierarchy Optimizationscited by 1
  22. 2026Token Encoding for Semantic Recoverycited by 1
  23. 2026SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inferencecited by 1
  24. 2026When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inferencecited by 1
  25. 2026Energy-Efficient On-Device RAG on a Mobile NPU: System Design and Benchmark on Snapdragon X Elitecited by 1
  26. 2026Efficient On-Device Diffusion LLM Inference with Mobile NPUcited by 1
  27. 2025Bridging the Digital Divide: Small Language Models as a Pathway for Physics and Photonics Education in Underdeveloped Regionscited by 0
  28. 2025Context-Driven Performance Modeling for Causal Inference Operators on Neural Processing Unitscited by 0
  29. 2025Parallax: Runtime Parallelization for Operator Fallbacks in Heterogeneous Edge Systemscited by 0
  30. 2025Atom: Efficient On-Device Video-Language Pipelines Through Modular Reusecited by 0
  31. 2026TriGen: NPU Architecture for End-to-End Acceleration of Large Language Models based on SW-HW Co-Designcited by 0
  32. 2026ANVIL: Accelerator-Native Video Interpolation via Codec Motion Vector Priorscited by 0
  33. 2026Make Your LVLM KV Cache More Lightweightcited by 0
  34. 2026TileFuse: A Fused Mixed-Precision Kernel Library for Efficient Quantized LLM Inference on AMD NPUscited by 0
  35. 2026Less is More: Lightweight Prompt Compression for Question Answering Applications on Edge Devicescited by 0
  36. 2026Continual Learning for Sequential Personalization of Small Language Models: A Stability Monitoring Analysiscited by 0
  37. 2026PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inferencecited by 0
  38. 2026Beyond Prefill-Decode Disaggregation: Dissecting LLM Inference for Heterogeneous Platforms via Dynamic Operator Schedulingcited by 0

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference
  2. 2023Efficient LLM inference solution on Intel GPU
  3. 2025Scaling LLM Test-Time Compute with Mobile NPU on Smartphones
  4. 2025Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference
  5. 2026EdgeFlow: Fast Cold Starts for LLMs on Mobile Devices ↗
  6. 2023Efficient LLM Inference on CPUs
  7. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory
  8. 2025MNN-LLM: A Generic Inference Engine for Fast Large Language Model Deployment on Mobile Devices