mlsanityData to 14 Oct 2026

HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Models

arXiv 2505.02795 ↗First posted 5 May 2025cs.LG cs.AI cs.DCcited by 31

Not on the map. Why?

Abstract (arXiv)

Recently, large language models (LLMs) have achieved remarkable breakthroughs, revolutionizing the natural language processing domain and beyond. Due to immense parameter sizes, fine-tuning these models with private data for diverse downstream tasks has become mainstream. Though federated learning (FL) offers a promising solution for fine-tuning LLMs without sharing raw data, substantial computing costs hinder its democratization. Moreover, in real-world scenarios, private client devices often possess heterogeneous computing resources, further complicating LLM fine-tuning. To combat these challenges, we propose HSplitLoRA, a heterogeneous parameter-efficient fine-tuning (PEFT) framework built on split learning (SL) and low-rank adaptation (LoRA) fine-tuning, for efficiently fine-tuning LLMs on heterogeneous client devices. HSplitLoRA first identifies important weights based on their contributions to LLM training. It then dynamically configures the decomposition ranks of LoRA adapters for selected weights and determines the model split point according to varying computing budgets of client devices. Finally, a noise-free adapter aggregation mechanism is devised to support heterogeneous adapter aggregation without introducing noise. Extensive experiments demonstrate that HSplitLoRA outperforms state-of-the-art benchmarks in training accuracy and convergence speed.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 56 of its 78 references that are in the map. First posted on arXiv 5 May 2025.

Builds on

56 of 78 references.

References · 56

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  7. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2023BloombergGPT: A Large Language Model for Financecited by 939
  11. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  12. 2020What is the State of Neural Network Pruning?cited by 583
38 more
  1. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  2. 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 395
  3. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 377
  4. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  5. 2020SplitFed: When Federated Learning Meets Split Learningcited by 310
  6. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  7. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  8. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  9. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  10. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  11. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  12. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
  13. 2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 141
  14. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  15. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  16. 2023Efficient Parallel Split Learning over Resource-constrained Wireless Edge Networkscited by 95
  17. 2018Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?cited by 94
  18. 2023Pushing Large Language Models to the 6G Edge: Vision, Challenges, and Opportunitiescited by 90
  19. 2022Decentralized Training of Foundation Models in Heterogeneous Environmentscited by 87
  20. 2022Split Learning over Wireless Networks: Parallel Design and Resource Managementcited by 75
  21. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
  22. 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
  23. 2023Split Learning in 6G Edge Networkscited by 68
  24. 2023Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimizationcited by 65
  25. 2024AdaptSFL: Adaptive Split Federated Learning in Resource-constrained Edge Networkscited by 64
  26. 2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 62
  27. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
  28. 2022FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clientscited by 46
  29. 2022Two-stage LLM Fine-tuning with Less Specialization and More Generalizationcited by 41
  30. 2024AgentsCoDriver: Large Language Model Empowered Collaborative Driving with Lifelong Learningcited by 40
  31. 2024LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluationcited by 38
  32. 2024Hierarchical Split Federated Learning: Convergence Analysis and System Optimizationcited by 38
  33. 2023TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Trainingcited by 27
  34. 2021Server-Side Local Gradient Averaging and Learning Rate Acceleration for Scalable Split Learningcited by 23
  35. 2022A Bargaining Game for Personalized, Energy Efficient Split Learning over Wireless Networkscited by 22
  36. 2024CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inferencecited by 22
  37. 2025LEO-Split: A Semi-Supervised Split Learning Framework over LEO Satellite Networkscited by 22
  38. 2024SatFed: A Resource-Efficient LEO Satellite-Assisted Heterogeneous Federated Learning Frameworkcited by 21

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 16.

31 papers.

Sort:

By area:

  1. 2025HASFL: Heterogeneity-aware Split Federated Learning over Edge Computing Systemscited by 15
  2. 2026HFedMoE: Resource-aware Heterogeneous Federated Learning with Mixture-of-Expertscited by 11
  3. 2025Dynamic Uncertainty-aware Multimodal Fusion for Outdoor Health Monitoringcited by 10
  4. 2025LLM-Driven Stationarity-Aware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systemscited by 8
  5. 2025SlimCaching: Edge Caching of Mixture-of-Experts for Distributed Inferencecited by 7
  6. 2025LEED: A Highly Efficient and Scalable LLM-Empowered Expert Demonstrations Framework for Multi-Agent Reinforcement Learningcited by 6
  7. 2026Conflict-Aware Client Selection for Multi-Server Federated Learningcited by 5
  8. 2026Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneitycited by 4
  9. 2025Visual Fidelity Index for Generative Semantic Communications with Critical Information Embeddingcited by 3
  10. 2025Defensive Adversarial CAPTCHA: A Semantics-Driven Framework for Natural Adversarial Example Generationcited by 3
  11. 2025A Satellite-Ground Synergistic Large Vision-Language Model System for Earth Observationcited by 3
  12. 2025Sample Efficient Experience Replay in Non-stationary Environmentscited by 3
19 more
  1. 2026HOSL: Hybrid-Order Split Learning for Memory-Constrained Edge Trainingcited by 3
  2. 2025Adaptive and Resource-efficient Agentic AI Systems for Mobile and Embedded Devices: A Surveycited by 2
  3. 2025Intra-DP: A High Performance Collaborative Inference System for Mobile Edge Computingcited by 1
  4. 2025PrivTune: Efficient and Privacy-Preserving Fine-Tuning of Large Language Models via Device-Cloud Collaborationcited by 1
  5. 2025Certifying the Right to Be Forgotten: Primal-Dual Optimization for Sample and Label Unlearning in Vertical Federated Learningcited by 1
  6. 2026SL-FAC: A Communication-Efficient Split Learning Framework with Frequency-Aware Compressioncited by 1
  7. 2026Physically-Induced Atmospheric Adversarial Perturbations: Enhancing Transferability and Robustness in Remote Sensing Image Classificationcited by 1
  8. 2026AutoEncoder-Compressed Parallel Split Learning for Pre-trained Model Fine-Tuningcited by 1
  9. 2025DP-TRAE: A Dual-Phase Merging Transferable Reversible Adversarial Example for Image Privacy Protectioncited by 0
  10. 2025HiCoLoRA: Addressing Context-Prompt Misalignment via Hierarchical Collaborative LoRA for Zero-Shot DSTcited by 0
  11. 2026Exploiting Adaptive Channel Pruning for Communication-Efficient Split Learningcited by 0
  12. 2026GAPSL: A Gradient-Aligned Parallel Split Learning over Data-Heterogeneous Edge Computing Systemscited by 0
  13. 2026A Survey on Split Learning for LLM Fine-Tuning: Models, Systems, and Privacy Optimizationscited by 0
  14. 2026Conflict-Aware Federated Fine-Tuning of Large Language Models with Mixture-of-Expertscited by 0
  15. 2026Large Language Model Enhanced Differentiable Trajectory Planning for IoT-Enabled Autonomous Drivingcited by 0
  16. 2026When Agentic AI Meets Integrated Sensing and Communicationcited by 0
  17. 2026Unveiling the Depth-Performance Dilemma in Split-Federated Fine-tuning of LLMscited by 0
  18. 2026LLM-Based Schema-Aware Split Learning for Privacy-Preserving Mental Distress Prediction Across Heterogeneous Surveyscited by 0
  19. 2026DeCo: Efficient Decouple-to-Couple Learning for Multi-Task Visual Groundingcited by 0