# Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devices

arXiv: https://arxiv.org/abs/2406.03777 · HTML: https://mlsanity.com/paper/2406.03777/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 83 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 30 of 62)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)

## Built on by (most cited first, 17 of 17)

- [Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency](https://mlsanity.com/paper/2504.03360/) (2025)
- [Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2503.16585/) (2025)
- [PEANuT: Parameter-Efficient Adaptation with Weight-aware Neural Tweakers](https://mlsanity.com/paper/2410.01870/) (2024)
- [Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologies](https://mlsanity.com/paper/2410.08860/) (2024)
- [Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models](https://mlsanity.com/paper/2508.17184/) (2025)
- [Automatic Screening for Children with Speech Disorder using Automatic Speech Recognition: Opportunities and Challenges](https://mlsanity.com/paper/2410.11865/) (2024)
- [Tiny-Align: Bridging Automatic Speech Recognition and Large Language Model on the Edge](https://mlsanity.com/paper/2411.13766/) (2024)
- [Harnessing the Potential of Large Language Models in Modern Marketing Management: Applications, Future Directions, and Strategic Recommendations](https://mlsanity.com/paper/2501.10685/) (2025)
- [EdgeProfiler: A Fast Profiling Framework for Lightweight LLMs on Edge Using Analytical Model](https://mlsanity.com/paper/2506.09061/) (2025)
- [NVCiM-PT: An NVCiM-assisted Prompt Tuning Framework for Edge LLMs](https://mlsanity.com/paper/2411.08244/) (2024)
- [Biases in Edge Language Models: Detection, Analysis, and Mitigation](https://mlsanity.com/paper/2502.11349/) (2025)
- [TaleFrame: An Interactive Story Generation System with Fine-Grained Control and Large Language Models](https://mlsanity.com/paper/2512.02402/) (2025)
- [CiMRAG: CiM-Aware Domain-Adaptive and Noise-Resilient Retrieval-Augmented Generation for Edge-Based LLMs](https://mlsanity.com/paper/2601.20041/) (2026)
- [CQ-CiM: Hardware-Aware Embedding Shaping for Robust CiM-Based Retrieval](https://mlsanity.com/paper/2602.20083/) (2026)
- [Distilling Safe LLM Systems via Soft Prompts for On Device Settings](https://mlsanity.com/paper/2606.09388/) (2026)
- [Charting the Growth of Social-Physical HRI (spHRI): A Systematic Review Pipeline Augmented by Small Language Models](https://mlsanity.com/paper/2606.26382/) (2026)
- [Towards Secure Cloud-Native Computing: Unveiling Kubernetes Misconfigurations with Large Language Models](https://mlsanity.com/paper/2609.20834/) (2026)

## Abstract (arXiv)

The scaling laws have become the de facto guidelines for designing large language models (LLMs), but they were studied under the assumption of unlimited computing resources for both training and inference. As LLMs are increasingly used as personalized intelligent assistants, their customization (i.e., learning through fine-tuning) and deployment onto resource-constrained edge devices will become more and more prevalent. An urging but open question is how a resource-constrained computing environment would affect the design choices for a personalized LLM. We study this problem empirically in this work. In particular, we consider the tradeoffs among a number of key design factors and their intertwined impacts on learning efficiency and accuracy. The factors include the learning methods for LLM customization, the amount of personalized data used for learning customization, the types and sizes of LLMs, the compression methods of LLMs, the amount of time afforded to learn, and the difficulty levels of the target use cases. Through extensive experimentation and benchmarking, we draw a number of surprisingly insightful guidelines for deploying LLMs onto resource-constrained devices. For example, an optimal choice between parameter learning and RAG may vary depending on the difficulty of the downstream task, the longer fine-tuning time does not necessarily help the model, and a compressed LLM may be a better choice than an uncompressed LLM to learn from limited personalized data.