mlsanityData to 7 Oct 2026

When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference

arXiv 2605.27435 ↗First posted 22 May 2026cs.AR cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Deploying large language models (LLMs) on mobile devices increasingly relies on heterogeneous execution, yet no prior study has systematically characterized NPU effectiveness at the operator and pipeline level. We present the first stage-aware, multi-level benchmarking study of mobile LLM inference on a CPU-NPU heterogeneous SoC. We introduce an OPMASK-based controlled pipeline decomposition methodology that isolates communication, quantization, and computation overheads within the NPU execution path. Our results reveal a counter-intuitive stage-level performance reversal: CPUs outperform NPUs in the compute-intensive Prefill stage (up to 1.6x), while NPUs provide only limited acceleration in the memory-bound Decode stage (1.05-1.2x). We further show that scheduling overhead and cross-backend fallback reduce the practical benefits of NPU offloading. For the energy trend, increasing NPU offloading leads to higher energy consumption (up to 51%). Based on these findings, we derive design guidelines for NPU architects targeting on-device LLM inference.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 19 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

14 of 19 references.

References · 14

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  6. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
  7. 2024Scaling Laws for Precisioncited by 92
  8. 2024FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGAscited by 60
  9. 2024Fast On-device LLM Inference with NPUscited by 58
  10. 2025Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Modelscited by 44
  11. 2025Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inferencecited by 20
  12. 2025Scaling LLM Test-Time Compute with Mobile NPU on Smartphonescited by 9
2 more
  1. 2025A Survey of LLM Inference Systemscited by 8
  2. 2025Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inferencecited by 3

Built on by

Not cited yet.