mlsanityData to 14 Oct 2026

FedLoDrop: Federated LoRA with Dropout for Generalized LLM Fine-tuning

arXiv 2510.12078 ↗First posted 14 Oct 2025cs.IT cs.LG math.ITcited by 2

Not on the map. Why?

Abstract (arXiv)

Fine-tuning (FT) large language models (LLMs) is crucial for adapting general-purpose models to specific tasks, enhancing accuracy and relevance with minimal resources. To further enhance generalization ability while reducing training costs, this paper proposes Federated LoRA with Dropout (FedLoDrop), a new framework that applies dropout to the rows and columns of the trainable matrix in Federated LoRA. A generalization error bound and convergence analysis under sparsity regularization are obtained, which elucidate the fundamental trade-off between underfitting and overfitting. The error bound reveals that a higher dropout rate increases model sparsity, thereby lowering the upper bound of pointwise hypothesis stability (PHS). While this reduces the gap between empirical and generalization errors, it also incurs a higher empirical error, which, together with the gap, determines the overall generalization error. On the other hand, though dropout reduces communication costs, deploying FedLoDrop at the network edge still faces challenges due to limited network resources. To address this issue, an optimization problem is formulated to minimize the upper bound of the generalization error, by jointly optimizing the dropout rate and resource allocation subject to the latency and per-device energy consumption constraints. To solve this problem, a branch-and-bound (B&B)-based method is proposed to obtain its globally optimal solution. Moreover, to reduce the high computational complexity of the B&B-based method, a penalized successive convex approximation (P-SCA)-based algorithm is proposed to efficiently obtain its high-quality suboptimal solution. Finally, numerical results demonstrate the effectiveness of the proposed approach in mitigating overfitting and improving the generalization capability.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 14 Oct 2025.

Builds on

29 of 45 references.

References · 29

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2022Prompt-aligned Gradient for Prompt Tuningcited by 341
  3. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  4. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  5. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 141
  6. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 136
  7. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
  8. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  9. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  10. 2017Stability and Generalization of Learning Algorithms that Converge to Global Optimacited by 100
  11. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 93
  12. 2021Federated Dropout -- A Simple Approach for Enabling Federated Learning on Resource Constrained Devicescited by 56
17 more
  1. 2023Large Language Models Empowered Autonomous Edge AI for Connected Intelligencecited by 56
  2. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  3. 2022Transformers from an Optimization Perspectivecited by 27
  4. 2024Federated LoRA with Sparse Communicationcited by 26
  5. 2024Federated Fine-Tuning for Pre-Trained Foundation Models Over Wireless Networkscited by 25
  6. 2024FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Modelscited by 18
  7. 2023Spectrum Breathing: Protecting Over-the-Air Federated Learning Against Interferencecited by 9
  8. 2023Semantic Communications using Foundation Models: Design Approaches and Open Issuescited by 9
  9. 2020Adaptive Subcarrier, Parameter, and Power Allocation for Partitioned Edge Learning Over Broadband Channelscited by 5
  10. 2023KnowPrefix-Tuning: A Two-Stage Prefix-Tuning Framework for Knowledge-Grounded Dialogue Generationcited by 5
  11. 2025Integrated Sensing, Communication, and Computation for Over-the-Air Federated Edge Learningcited by 5
  12. 2024Resource Management for Low-latency Cooperative Fine-tuning of Foundation Models at the Network Edgecited by 4
  13. 2024DEeR: Deviation Eliminating and Noise Regulating for Privacy-preserving Federated Low-rank Adaptationcited by 4
  14. 2025Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Modelscited by 4
  15. 2025Resource-Efficient Federated Fine-Tuning Large Language Models for Heterogeneous Datacited by 4
  16. 2025Federated Intelligence: When Large AI Models Meet Federated Fine-Tuning and Collaborative Reasoning at the Network Edgecited by 4
  17. 2024MIRA: A Method of Federated MultI-Task Learning for LaRge LAnguage Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Graph Representation Learning Augmented Model Manipulation on Federated Fine-Tuning of LLMscited by 1
  2. 2026FED-FSTQ: Fisher-Guided Token Quantization for Communication-Efficient Federated Fine-Tuning of LLMs on Edge Devicescited by 0