# On-Device LLMs for SMEs: Challenges and Opportunities

arXiv: https://arxiv.org/abs/2410.16070 · HTML: https://mlsanity.com/paper/2410.16070/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 29 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Builds on (most cited first, 14 of 14)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression](https://mlsanity.com/paper/2306.03078/) (2023)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better](https://mlsanity.com/paper/2106.08962/) (2021)
- [Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications](https://mlsanity.com/paper/2111.12444/) (2021)
- [AIOS: LLM Agent Operating System](https://mlsanity.com/paper/2403.16971/) (2024)
- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models](https://mlsanity.com/paper/2401.00625/) (2024)
- [LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report](https://mlsanity.com/paper/2405.00732/) (2024)
- [Accurate LoRA-Finetuning Quantization of LLMs via Information Retention](https://mlsanity.com/paper/2402.05445/) (2024)
- [Optimizing LLM Queries in Relational Data Analytics Workloads](https://mlsanity.com/paper/2403.05821/) (2024)
- [Retrieval Augmented End-to-End Spoken Dialog Models](https://mlsanity.com/paper/2402.01828/) (2024)
- [Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models](https://mlsanity.com/paper/2310.05015/) (2023)
- [eDKM: An Efficient and Accurate Train-time Weight Clustering for Large Language Models](https://mlsanity.com/paper/2309.00964/) (2023)

## Built on by (most cited first, 2 of 2)

- [LLMs Have Rhythm: Fingerprinting Large Language Models Using Inter-Token Times and Network Traffic Analysis](https://mlsanity.com/paper/2502.20589/) (2025)
- [EdgeProfiler: A Fast Profiling Framework for Lightweight LLMs on Edge Using Analytical Model](https://mlsanity.com/paper/2506.09061/) (2025)

## Abstract (arXiv)

This paper presents a systematic review of the infrastructure requirements for deploying Large Language Models (LLMs) on-device within the context of small and medium-sized enterprises (SMEs), focusing on both hardware and software perspectives. From the hardware viewpoint, we discuss the utilization of processing units like GPUs and TPUs, efficient memory and storage solutions, and strategies for effective deployment, addressing the challenges of limited computational resources typical in SME settings. From the software perspective, we explore framework compatibility, operating system optimization, and the use of specialized libraries tailored for resource-constrained environments. The review is structured to first identify the unique challenges faced by SMEs in deploying LLMs on-device, followed by an exploration of the opportunities that both hardware innovations and software adaptations offer to overcome these obstacles. Such a structured review provides practical insights, contributing significantly to the community by enhancing the technological resilience of SMEs in integrating LLMs.