# LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load

arXiv: https://arxiv.org/abs/2603.23640 · HTML: https://mlsanity.com/paper/2603.23640/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 15 references that are in the map.
- First posted on arXiv 24 Mar 2026.

## Builds on (most cited first, 10 of 10)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [AI Benchmark: Running Deep Neural Networks on Android Smartphones](https://mlsanity.com/paper/1810.01109/) (2018)
- [MELTing point: Mobile Evaluation of Language Transformers](https://mlsanity.com/paper/2403.12844/) (2024)
- [Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs](https://mlsanity.com/paper/2403.20041/) (2024)
- [Understanding Large Language Models in Your Pockets: Performance Study on COTS Mobile Devices](https://mlsanity.com/paper/2410.03613/) (2024)
- [Dissecting the Impact of Mobile DVFS Governors on LLM Inference Performance and Energy Efficiency](https://mlsanity.com/paper/2507.02135/) (2025)
- [Challenging GPU Dominance: When CPUs Outperform for On-Device LLM Inference](https://mlsanity.com/paper/2505.06461/) (2025)

## Built on by (most cited first, 4 of 4)

- [Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers](https://mlsanity.com/paper/2604.24785/) (2026)
- [Efficient LLM Distillation for Bangladesh Legal Context: A Smartphone-Compatible Retrieval-Augmented Generation Model](https://mlsanity.com/paper/2609.24177/) (2026)
- [Fine-Tuning a 3B-Parameter LLM on a Smartphone: Characterizing Sustained Training](https://mlsanity.com/paper/2610.06325/) (2026)
- [Seeing at the Edge: Energy Characterization of Vision-Language Model Inference on Edge Platforms](https://mlsanity.com/paper/2610.20136/) (2026)

## Abstract (arXiv)

Deploying large language models on-device for always-on personal agents demands sustained inference from hardware tightly constrained in power, thermal envelope, and memory. We benchmark Qwen 2.5 1.5B (4-bit quantised) across four platforms: a Raspberry Pi 5 with Hailo-10H NPU, a Samsung Galaxy S24 Ultra, an iPhone 16 Pro, and a laptop NVIDIA RTX 4050 GPU. Using a fixed 258-token prompt over 20 warm-condition iterations per device, we measure throughput, latency, power, and thermal behaviour. For mobile platforms, thermal management supersedes peak compute as the primary constraint: the iPhone 16 Pro loses nearly half its throughput within two iterations, and the S24 Ultra suffers a hard OS-enforced GPU frequency floor that terminates inference entirely. On dedicated hardware, distinct constraints dominate: the RTX 4050 is bounded by its battery power ceiling, while the Hailo-10H is limited by on-module memory bandwidth. The RTX 4050 sustains 131.7 tok/s at 34.1 W; the Hailo-10H sustains 6.9 tok/s at under 2 W with near-zero variance, matching the RTX 4050 in energy proportionality at 19x lower throughput. Results should be interpreted as platform-level deployment characterisations for a single model and prompt type, reflecting hardware and software combined, rather than general claims about hardware capability alone.