# Cloud to Edge: Benchmarking LLM Inference On Hardware-Accelerated Single-Board Computers

arXiv: https://arxiv.org/abs/2604.24785 · HTML: https://mlsanity.com/paper/2604.24785/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 24 Apr 2026.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing](https://mlsanity.com/paper/2310.08433/) (2023)
- [A Review on Edge Large Language Models: Design, Execution, and Applications](https://mlsanity.com/paper/2410.11845/) (2024)
- [Interpreting and Improving Large Language Models in Arithmetic Calculation](https://mlsanity.com/paper/2409.01659/) (2024)
- [DeepEdgeBench: Benchmarking Deep Neural Networks on Edge Devices](https://mlsanity.com/paper/2108.09457/) (2021)
- [Edge AI: A Taxonomy, Systematic Review and Future Directions](https://mlsanity.com/paper/2407.04053/) (2024)
- [SoK: The Privacy Paradox of Large Language Models: Advancements, Privacy Risks, and Mitigation](https://mlsanity.com/paper/2506.12699/) (2025)
- [Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective](https://mlsanity.com/paper/2508.16712/) (2025)
- [Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring](https://mlsanity.com/paper/2501.13331/) (2025)
- [LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load](https://mlsanity.com/paper/2603.23640/) (2026)

## Built on by (most cited first, 1 of 1)

- [Benchmarking and Reasoning Distillation of Large Language Models for Feedback Controller Design in Complex Dynamical Systems](https://mlsanity.com/paper/2608.07004/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are becoming increasingly capable at small parameter scales. At the same time, conventional cloud-centric deployment introduces challenges around data privacy, latency, and cost that are acute in operational technology and defence environments. Advances in model distillation, quantisation, and affordable edge accelerators now make local LLM inference on single-board computers feasible, but the high dimensionality of the configuration space makes identifying optimal deployments difficult without structured evaluation. Existing LLM-specific edge benchmarking efforts rely on CPU-only inference, poor coverage of genuine single-board computers, and generic evaluation tasks that lack multi-dimensional assessment of hardware effectiveness. This paper proposes a multi-dimensional benchmarking methodology that jointly evaluates inference performance and hardware efficiency across four IoT-suitable edge platform configurations testing single-board computers with the latest available hardware accelerators. Our results reveal the benefits of using hardware accelerators such as NPUs and GPUs, along with multi-dimensional evaluations quantifying the trade-offs between power efficiency, physical device size and token throughput; offering practical guidance for deploying generative AI in privacy-sensitive and connectivity-limited environments such as unmanned vehicles and portable, ruggedised operations.