# SplitFT: An Adaptive Federated Split Learning System For LLMs Fine-Tuning

arXiv: https://arxiv.org/abs/2604.26388 · HTML: https://mlsanity.com/paper/2604.26388/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 51 references that are in the map.
- First posted on arXiv 29 Apr 2026.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [LoRA+: Efficient Low Rank Adaptation of Large Models](https://mlsanity.com/paper/2402.12354/) (2024)
- [SplitFed: When Federated Learning Meets Split Learning](https://mlsanity.com/paper/2004.12088/) (2020)
- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA](https://mlsanity.com/paper/2312.03732/) (2023)
- [Large language models in healthcare and medical domain: A review](https://mlsanity.com/paper/2401.06775/) (2023)
- [LoRA-Pro: Are Low-Rank Adapters Properly Optimized?](https://mlsanity.com/paper/2407.18242/) (2024)
- [TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection](https://mlsanity.com/paper/1911.04118/) (2019)
- [Fairness and Privacy-Preserving in Federated Learning: A Survey](https://mlsanity.com/paper/2306.08402/) (2023)
- [FedFa: A Fully Asynchronous Training Paradigm for Federated Learning](https://mlsanity.com/paper/2404.11015/) (2024)
- [Communication and Storage Efficient Federated Split Learning](https://mlsanity.com/paper/2302.05599/) (2023)
- [A Survey on Error-Bounded Lossy Compression for Scientific Datasets](https://mlsanity.com/paper/2404.02840/) (2024)
- [An Optimized Error-controlled MPI Collective Framework Integrated with Lossy Compression](https://mlsanity.com/paper/2304.03890/) (2023)
- [HLoRA: Efficient Federated Learning System for LLM Heterogeneous Fine-Tuning](https://mlsanity.com/paper/2503.00813/) (2025)
- [ZCCL: Significantly Improving Collective Communication With Error-Bounded Lossy Compression](https://mlsanity.com/paper/2502.18554/) (2025)
- [Theoretical Analysis of Privacy Leakage in Trustworthy Federated Learning: A Perspective from Linear Algebra and Optimization Theory](https://mlsanity.com/paper/2407.16735/) (2024)
- [CLLoRA: An Approach to Measure the Effects of the Context Length for LLM Fine-Tuning](https://mlsanity.com/paper/2502.18910/) (2025)
- [MoE-Compression: How the Compression Error of Experts Affects the Inference Accuracy of MoE Model?](https://mlsanity.com/paper/2509.07727/) (2025)
- [An Efficient Gradient-Aware Error-Bounded Lossy Compressor for Federated Learning](https://mlsanity.com/paper/2511.05770/) (2025)

## Built on by (most cited first, 1 of 1)

- [When Is Shallow Enough? Adaptive Split Federated Learning with Client-Specific Sufficiency Estimation](https://mlsanity.com/paper/2608.15639/) (2026)

## Abstract (arXiv)

Federated Split Learning has been identified as an efficient approach to address the computational resource constraints of clients in classical federated learning, while guaranteeing data privacy for distributed model training across data owners. However, it faces some critical challenges when such a training strategy meets large language models (LLMs) for fine-tuning. Such challenges include setting the cutlayer adaptively across different clients to address the data and device heterogeneity issues, which affect the system performance significantly. In addition, efficiently reducing the communication overhead during the fine-tuning procedure is also another challenge. No work tries to address these challenges. To bridge this gap, we propose SplitTF, an adaptive federated split learning system for LLMs fine-tuning. SplitFT enables different clients to set different cut layers according to their computation resources and trained model performance. SplitFT also proposes to reduce the LoRA rank in cutlayer to reduce the communication overhead. In addition to simulating the heterogeneous data in real-world applications for our proposed split federated learning system, we propose a length-based Dirichlet approach to divide the training data into different clients. Extensive experimental results show that our proposed approach outperforms the state-of-the-art approach for fine-tuning time efficiency and model performance based on various popular benchmarks.