# Low-Parameter Federated Learning with Large Language Models

arXiv: https://arxiv.org/abs/2307.13896 · HTML: https://mlsanity.com/paper/2307.13896/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2023.

## Built on by (most cited first, 23 of 23)

- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Mobile Edge Intelligence for Large Language Models: A Contemporary Survey](https://mlsanity.com/paper/2407.18921/) (2024)
- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes](https://mlsanity.com/paper/2312.06353/) (2023)
- [Dual-Personalizing Adapter for Federated Foundation Models](https://mlsanity.com/paper/2403.19211/) (2024)
- [The Future of Large Language Model Pre-training is Federated](https://mlsanity.com/paper/2405.10853/) (2024)
- [Federated Large Language Models: Current Progress and Future Directions](https://mlsanity.com/paper/2409.15723/) (2024)
- [Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Survey](https://mlsanity.com/paper/2406.05392/) (2024)
- [Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA](https://mlsanity.com/paper/2308.04679/) (2023)
- [Personalized Wireless Federated Learning for Large Language Models](https://mlsanity.com/paper/2404.13238/) (2024)
- [The Role of Federated Learning in a Wireless World with Foundation Models](https://mlsanity.com/paper/2310.04003/) (2023)
- [FedRDMA: Communication-Efficient Cross-Silo Federated LLM via Chunked RDMA Transmission](https://mlsanity.com/paper/2403.00881/) (2024)
- [DEeR: Deviation Eliminating and Noise Regulating for Privacy-preserving Federated Low-rank Adaptation](https://mlsanity.com/paper/2410.12926/) (2024)
- [Federated TrustChain: Blockchain-Enhanced LLM Training and Unlearning](https://mlsanity.com/paper/2406.04076/) (2024)
- [FedHL: Federated Learning for Heterogeneous Low-Rank Adaptation via Unbiased Aggregation](https://mlsanity.com/paper/2505.18494/) (2025)
- [SoK: Privacy-aware LLM in Healthcare: Threat Model, Privacy Techniques, Challenges and Recommendations](https://mlsanity.com/paper/2601.10004/) (2026)
- [Toward Efficient Membership Inference Attacks against Federated Large Language Models: A Projection Residual Approach](https://mlsanity.com/paper/2604.21197/) (2026)
- [Recent Advances in Federated Learning Driven Large Language Models: A Survey on Architecture, Performance, and Security](https://mlsanity.com/paper/2406.09831/) (2024)
- [A Hybrid Swarm Intelligence Approach for Optimizing Multimodal Large Language Models Deployment in Edge-Cloud-based Federated Learning Environments](https://mlsanity.com/paper/2502.10419/) (2025)
- [Efficient Split Federated Learning for Large Language Models over Communication Networks](https://mlsanity.com/paper/2504.14667/) (2025)
- [Aggregating Low Rank Adapters in Federated Fine-tuning](https://mlsanity.com/paper/2501.06332/) (2025)
- [NWaaS: A Non-Intrusive and Privacy-Preserving Watermarking-as-a-Service System with Adaptive Resource Scheduling](https://mlsanity.com/paper/2507.18036/) (2025)
- [Can LoRA Fusion Support Cross-Domain Tasks in Cloud-Edge Collaboration?](https://mlsanity.com/paper/2605.23913/) (2026)

## Abstract (arXiv)

We study few-shot Natural Language Understanding (NLU) tasks with Large Language Models (LLMs) in federated learning (FL) scenarios. It is a challenging task due to limited labeled data and communication capacities in FL, especially with mobile devices. Recent studies show LLMs can be prompted to perform few-shot NLU tasks like sentiment analysis and arithmetic reasoning. However, the huge sizes of LLMs result in high computation and communication costs, making classical FL schemes impractical. To address these challenges, we propose Low-Parameter Federated Learning (LP-FL). LP-FL combines few-shot prompt learning from LLMs with efficient communication and federating techniques. Our approach enables federated clients to assign soft labels to unlabeled data using gradually learned knowledge from the global model. Through iterative soft-label assigning, we continually expand the labeled set during the FL process. Additionally, to reduce computation and communication costs, LP-FL utilizes the Low-Rank Adaptation (LoRA) technique for compact learnable parameter construction, efficient local model fine-tuning, and affordable global model federation. LP-FL consistently outperforms Full-Parameter Federated Learning (FP-FL) in sentiment analysis tasks across various FL settings. Its resistance to overfitting allows LP-FL to equal or surpass centralized training in few-shot scenarios.