An Evaluation of LLMs Inference on Popular Single-board Computers
Not on the map. Why?
Abstract (arXiv)
The growing demand for on-device large language model (LLM) inference is driving interest in deploying lightweight, cost-effective AI solutions on edge hardware. Single-board computers (SBCs) such as the Raspberry Pi and Orange Pi offer a promising platform for localized, privacy-preserving inference-but remain underexplored in the context of LLM workloads. In this work, we benchmark the performance of 25 quantized open-source LLMs across three SBCs-Raspberry Pi 4, Raspberry Pi 5, and Orange Pi 5 Pro-using two inference runtimes: Ollama and Llamafile. We evaluate generation throughput, memory usage, and power consumption under varying CPU configurations, using multiple prompt types to simulate realistic workloads. Our results show that SBCs can reliably support models up to 1.5B parameters, with Llamafile achieving up to 4x higher throughput and 30-40% lower power usage than Ollama. We identify architecture-specific bottlenecks, highlight runtime-level trade-offs, and provide practical deployment recommendations. This study offers the first broad evaluation of LLM inference on SBCs, bridging the gap between high-performance language models and affordable edge computing.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 13 of its 32 references that are in the map. First posted on arXiv 20 Oct 2025.
Builds on
13 of 32 references.
References · 13
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023Mistral 7Bcited by 3,065
- 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
- 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
- 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
- 2023From Words to Watts: Benchmarking the Energy Costs of Large Language Model Inferencecited by 168
- 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
- 2023LLMCarbon: Modeling the end-to-end Carbon Footprint of Large Language Modelscited by 71
- 2023Norm Tweaking: High-performance Low-bit Quantization of Large Language Modelscited by 33
- 2024Hybrid Heterogeneous Clusters Can Lower the Energy Consumption of LLM Inference Workloadscited by 17
- 2023Text vectorization via transformer-based language models and n-gram perplexitiescited by 2
Built on by
Not cited yet.