mlsanityData to 14 Oct 2026

Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference

arXiv 2505.06461 ↗First posted 9 May 2025cs.DC cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

The common assumption in on-device AI is that GPUs, with their superior parallel processing, always provide the best performance for large language model (LLM) inference. In this work, we challenge this notion by empirically demonstrating that, under certain conditions, CPUs can outperform GPUs for LLM inference on mobile devices. Using a 1-billion-parameter LLM deployed via llama.cpp on the iPhone 15 Pro, we show that a CPU-only configuration (two threads, F16 precision) achieves 17 tokens per second, surpassing the 12.8 tokens per second obtained with GPU acceleration. We analyze the architectural factors driving this counterintuitive result, revealing that GPU memory transfer overhead and CPU thread optimization play a critical role. Furthermore, we explore the impact of thread oversubscription, quantization strategies, and hardware constraints, providing new insights into efficient on-device AI execution. Our findings challenge conventional GPU-first thinking, highlighting the untapped potential of optimized CPU inference and paving the way for smarter deployment strategies in mobile AI. However, fully explaining the observed CPU advantage remains difficult due to limited access to low-level profiling tools on iOS.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 9 May 2025.

Builds on

21 of 31 references.

References · 21

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  10. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 583
  11. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  12. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
9 more
  1. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  2. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 122
  3. 2024Small Language Models: Survey, Measurements, and Insightscited by 112
  4. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  5. 2024Stable LM 2 1.6B Technical Reportcited by 90
  6. 2020MNN: A Universal and Efficient Inference Enginecited by 65
  7. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  8. 2024Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUscited by 17
  9. 2024SlimLM: An Efficient Small Language Model for On-Device Document Assistancecited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Loadcited by 4
  2. 2026When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inferencecited by 1
  3. 2026SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktopscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study
  2. 2025Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference
  3. 2026Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference
  4. 2026Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference
  5. 2025Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling
  6. 2025Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling
  7. 2024Inference Performance Optimization for Large Language Models on CPUs
  8. 2026Characterizing Parallelism Strategies in LLM Inference: Fundamental Compute-Communication Trade-offs