# SplitFrozen: Split Learning with Device-side Model Frozen for Fine-Tuning LLM on Heterogeneous Resource-Constrained Devices

arXiv: https://arxiv.org/abs/2503.18986 · HTML: https://mlsanity.com/paper/2503.18986/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 23 Mar 2025.

## Builds on (most cited first, 12 of 12)

- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [SplitFed: When Federated Learning Meets Split Learning](https://mlsanity.com/paper/2004.12088/) (2020)
- [Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunities](https://mlsanity.com/paper/2309.16739/) (2023)
- [FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Models](https://mlsanity.com/paper/2310.10049/) (2023)
- [SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Models](https://mlsanity.com/paper/2407.00952/) (2024)
- [Transformer Layers as Painters](https://mlsanity.com/paper/2407.09298/) (2024)
- [Large Wireless Model (LWM): A Foundation Model for Wireless Channels](https://mlsanity.com/paper/2411.08872/) (2024)
- [Federated Fine-Tuning for Pre-Trained Foundation Models Over Wireless Networks](https://mlsanity.com/paper/2407.02924/) (2024)
- [Personalized Wireless Federated Learning for Large Language Models](https://mlsanity.com/paper/2404.13238/) (2024)
- [MobiLLM: Enabling LLM Fine-Tuning on the Mobile Device via Server Assisted Side Tuning](https://mlsanity.com/paper/2502.20421/) (2025)
- [Snake Learning: A Communication- and Computation-Efficient Distributed Learning Framework for 6G](https://mlsanity.com/paper/2405.03372/) (2024)
- [Energy-Efficient Split Learning for Fine-Tuning Large Language Models in Edge Networks](https://mlsanity.com/paper/2412.00090/) (2024)

## Built on by (most cited first, 3 of 3)

- [AutoEncoder-Compressed Parallel Split Learning for Pre-trained Model Fine-Tuning](https://mlsanity.com/paper/2607.17913/) (2026)
- [Hetero-SplitEE: Split Learning of Neural Networks with Early Exits for Heterogeneous IoT Devices](https://mlsanity.com/paper/2512.09313/) (2025)
- [Unveiling the Depth-Performance Dilemma in Split-Federated Fine-tuning of LLMs](https://mlsanity.com/paper/2608.22188/) (2026)

## Abstract (arXiv)

Fine-tuning large language models (LLMs) on private, on-device data can empower tailored personalized AI agents. However, fine-tuning LLMs on resource-constrained edge devices faces significant challenges, including excessive computation overhead, device heterogeneity, and data imbalance. This paper proposes SplitFrozen, a split learning framework that enables efficient LLM fine-tuning by strategically freezing device-side model layers while centralizing parameter-efficient fine-tuning on the server. Our framework partitions LLMs into device-side frozen layers and server-side fine-tuning layers, where heterogeneous resource-constrained devices execute only forward propagation. To minimize server-side training costs, we integrate Low-Rank Adaptation (LoRA) into the server-side layers. A pipeline parallelism strategy further optimizes training efficiency by decoupling device-server computations and leveraging decomposed backward propagation. Experiments on GPT-2 with the MRPC, MNLI-matched, and SST-2 datasets demonstrate that SplitFrozen outperforms FedLoRA and SplitLoRA by 69.4% model accuracy under extremely imbalanced data, while reducing up to 86.8% device-side computations and 50.2% total training time. Experiments also validate the scalability of SplitFrozen on content generation task using Llama-3.2 model on GSM8K dataset.