# Fine-Tuning and Deploying Large Language Models Over Edges: Issues and Approaches

arXiv: https://arxiv.org/abs/2408.10691 · HTML: https://mlsanity.com/paper/2408.10691/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 19 references that are in the map.
- First posted on arXiv 20 Aug 2024.

## Builds on (most cited first, 15 of 15)

- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [Fine-Tuning Language Models with Just Forward Passes](https://mlsanity.com/paper/2305.17333/) (2023)
- [Towards Building the Federated GPT: Federated Instruction Tuning](https://mlsanity.com/paper/2305.05644/) (2023)
- [Distributed Learning in Wireless Networks: Recent Progress and Future Challenges](https://mlsanity.com/paper/2104.02151/) (2021)
- [When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods](https://mlsanity.com/paper/2212.10025/) (2022)
- [Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization](https://mlsanity.com/paper/2310.15080/) (2023)
- [Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes](https://mlsanity.com/paper/2312.06353/) (2023)
- [When Large Language Model Agents Meet 6G Networks: Perception, Grounding, and Alignment](https://mlsanity.com/paper/2401.07764/) (2024)
- [Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs](https://mlsanity.com/paper/2304.14999/) (2023)
- [FwdLLM: Efficient FedLLM using Forward Gradient](https://mlsanity.com/paper/2308.13894/) (2023)
- [DPZero: Private Fine-Tuning of Language Models without Backpropagation](https://mlsanity.com/paper/2310.09639/) (2023)
- [Just One Byte (per gradient): A Note on Low-Bandwidth Decentralized Language Model Finetuning Using Shared Randomness](https://mlsanity.com/paper/2306.10015/) (2023)
- [Device-Edge Cooperative Fine-Tuning of Foundation Models as a 6G Service](https://mlsanity.com/paper/2310.18602/) (2023)

## Built on by (most cited first, 6 of 6)

- [On the Crucial Role of Initialization for Matrix Factorization](https://mlsanity.com/paper/2410.18965/) (2024)
- [The Larger the Merrier? Efficient Large AI Model Inference in Wireless Edge Networks](https://mlsanity.com/paper/2505.09214/) (2025)
- [KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation](https://mlsanity.com/paper/2410.20777/) (2024)
- [A Hybrid Swarm Intelligence Approach for Optimizing Multimodal Large Language Models Deployment in Edge-Cloud-based Federated Learning Environments](https://mlsanity.com/paper/2502.10419/) (2025)
- [Advancing Model Refinement: Muon-Optimized Distillation and Quantization for LLM Deployment](https://mlsanity.com/paper/2601.09865/) (2026)
- [WaDi: Weight Direction-aware Distillation for One-step Image Synthesis](https://mlsanity.com/paper/2603.08258/) (2026)

## Abstract (arXiv)

Since the release of GPT2-1.5B in 2019, the large language models (LLMs) have evolved from specialized deep models to versatile foundation models. While demonstrating remarkable zero-shot ability, the LLMs still require fine-tuning on local datasets and substantial memory for deployment over the network edges. Traditional first-order fine-tuning techniques require significant GPU memory that exceeds the capacity of mainstream hardware. Besides, the LLMs have been expanded beyond text generation to create images, audio, video, and multi-modal content, necessitating careful investigation of efficient deployment strategies for large-scale foundation models. In response to these challenges, model fine-tuning and model-compression techniques have been developed to support the sustainable growth of LLMs by reducing both operational and capital expenditures. In this work, we provide a comprehensive overview of prevalent memory-efficient fine-tuning methods for deployment at the network edge. We also review state-of-the-art literature on model compression, offering insights into the deployment of LLMs at network edges.