# FedLoDrop: Federated LoRA with Dropout for Generalized LLM Fine-tuning

arXiv: https://arxiv.org/abs/2510.12078 · HTML: https://mlsanity.com/paper/2510.12078/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Builds on (most cited first, 29 of 29)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Prompt-aligned Gradient for Prompt Tuning](https://mlsanity.com/paper/2205.14865/) (2022)
- [Towards Building the Federated GPT: Federated Instruction Tuning](https://mlsanity.com/paper/2305.05644/) (2023)
- [Improving LoRA in Privacy-preserving Federated Learning](https://mlsanity.com/paper/2403.12313/) (2024)
- [On the Effectiveness of Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2211.15583/) (2022)
- [FedScale: Benchmarking Model and System Performance of Federated Learning at Scale](https://mlsanity.com/paper/2105.11367/) (2021)
- [OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning](https://mlsanity.com/paper/2402.06954/) (2024)
- [Mobile Edge Intelligence for Large Language Models: A Contemporary Survey](https://mlsanity.com/paper/2407.18921/) (2024)
- [Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models](https://mlsanity.com/paper/2401.06432/) (2024)
- [Stability and Generalization of Learning Algorithms that Converge to Global Optima](https://mlsanity.com/paper/1710.08402/) (2017)
- [SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models](https://mlsanity.com/paper/2308.06522/) (2023)
- [Federated Dropout -- A Simple Approach for Enabling Federated Learning on Resource Constrained Devices](https://mlsanity.com/paper/2109.15258/) (2021)
- [Large Language Models Empowered Autonomous Edge AI for Connected Intelligence](https://mlsanity.com/paper/2307.02779/) (2023)
- [LoRA Dropout as a Sparsity Regularizer for Overfitting Control](https://mlsanity.com/paper/2404.09610/) (2024)
- [Transformers from an Optimization Perspective](https://mlsanity.com/paper/2205.13891/) (2022)
- [Federated LoRA with Sparse Communication](https://mlsanity.com/paper/2406.05233/) (2024)
- [Federated Fine-Tuning for Pre-Trained Foundation Models Over Wireless Networks](https://mlsanity.com/paper/2407.02924/) (2024)
- [FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Models](https://mlsanity.com/paper/2410.09432/) (2024)
- [Spectrum Breathing: Protecting Over-the-Air Federated Learning Against Interference](https://mlsanity.com/paper/2305.05933/) (2023)
- [Semantic Communications using Foundation Models: Design Approaches and Open Issues](https://mlsanity.com/paper/2309.13315/) (2023)
- [Adaptive Subcarrier, Parameter, and Power Allocation for Partitioned Edge Learning Over Broadband Channels](https://mlsanity.com/paper/2010.04061/) (2020)
- [KnowPrefix-Tuning: A Two-Stage Prefix-Tuning Framework for Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2306.15430/) (2023)
- [Integrated Sensing, Communication, and Computation for Over-the-Air Federated Edge Learning](https://mlsanity.com/paper/2508.15185/) (2025)
- [Resource Management for Low-latency Cooperative Fine-tuning of Foundation Models at the Network Edge](https://mlsanity.com/paper/2407.09873/) (2024)
- [DEeR: Deviation Eliminating and Noise Regulating for Privacy-preserving Federated Low-rank Adaptation](https://mlsanity.com/paper/2410.12926/) (2024)
- [Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models](https://mlsanity.com/paper/2501.14406/) (2025)
- [Resource-Efficient Federated Fine-Tuning Large Language Models for Heterogeneous Data](https://mlsanity.com/paper/2503.21213/) (2025)
- [Federated Intelligence: When Large AI Models Meet Federated Fine-Tuning and Collaborative Reasoning at the Network Edge](https://mlsanity.com/paper/2503.21412/) (2025)
- [MIRA: A Method of Federated MultI-Task Learning for LaRge LAnguage Models](https://mlsanity.com/paper/2410.15524/) (2024)

## Built on by (most cited first, 2 of 2)

- [Graph Representation Learning Augmented Model Manipulation on Federated Fine-Tuning of LLMs](https://mlsanity.com/paper/2605.07961/) (2026)
- [FED-FSTQ: Fisher-Guided Token Quantization for Communication-Efficient Federated Fine-Tuning of LLMs on Edge Devices](https://mlsanity.com/paper/2604.25421/) (2026)

## Abstract (arXiv)

Fine-tuning (FT) large language models (LLMs) is crucial for adapting general-purpose models to specific tasks, enhancing accuracy and relevance with minimal resources. To further enhance generalization ability while reducing training costs, this paper proposes Federated LoRA with Dropout (FedLoDrop), a new framework that applies dropout to the rows and columns of the trainable matrix in Federated LoRA. A generalization error bound and convergence analysis under sparsity regularization are obtained, which elucidate the fundamental trade-off between underfitting and overfitting. The error bound reveals that a higher dropout rate increases model sparsity, thereby lowering the upper bound of pointwise hypothesis stability (PHS). While this reduces the gap between empirical and generalization errors, it also incurs a higher empirical error, which, together with the gap, determines the overall generalization error. On the other hand, though dropout reduces communication costs, deploying FedLoDrop at the network edge still faces challenges due to limited network resources. To address this issue, an optimization problem is formulated to minimize the upper bound of the generalization error, by jointly optimizing the dropout rate and resource allocation subject to the latency and per-device energy consumption constraints. To solve this problem, a branch-and-bound (B&B)-based method is proposed to obtain its globally optimal solution. Moreover, to reduce the high computational complexity of the B&B-based method, a penalized successive convex approximation (P-SCA)-based algorithm is proposed to efficiently obtain its high-quality suboptimal solution. Finally, numerical results demonstrate the effectiveness of the proposed approach in mitigating overfitting and improving the generalization capability.