mlsanityData to 7 Oct 2026

Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models

arXiv 2410.13097 ↗First posted 16 Oct 2024cs.LG cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Parameter-efficient fine-tuning (PEFT) methods typically assume that Large Language Models (LLMs) are trained on data from a single device or client. However, real-world scenarios often require fine-tuning these models on private data distributed across multiple devices. Federated Learning (FL) offers an appealing solution by preserving user privacy, as sensitive data remains on local devices during training. Nonetheless, integrating PEFT methods into FL introduces two main challenges: communication overhead and data heterogeneity. In this paper, we introduce FedTT and FedTT+, methods for adapting LLMs by integrating tensorized adapters into client-side models' encoder/decoder blocks. FedTT is versatile and can be applied to both cross-silo FL and large-scale cross-device FL. FedTT+, an extension of FedTT tailored for cross-silo FL, enhances robustness against data heterogeneity by adaptively freezing portions of tensor factors, further reducing the number of trainable parameters. Experiments on BERT and LLaMA models demonstrate that our proposed methods successfully address data heterogeneity challenges and perform on par or even better than existing federated PEFT approaches while achieving up to 10× reduction in communication cost.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 55 of its 69 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

55 of 69 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2016Deep Learning with Differential Privacycited by 2,682
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  13. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  14. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  15. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  16. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  17. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  18. 2015Tensorizing Neural Networkscited by 246
  19. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  20. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 144
  21. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
  22. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 114
  23. 2024Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resourcescited by 93
  24. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  25. 2022FedPrompt: Communication-Efficient and Privacy Preserving Prompt Tuning in Federated Learningcited by 86
  26. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  27. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  28. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  29. 2023Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimizationcited by 65
  30. 2023FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Modelscited by 62
  31. 2023Parameter-Efficient Fine-Tuning without Introducing New Latencycited by 59
  32. 2020Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneitycited by 56
  33. 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
  34. 2024Federated LoRA with Sparse Communicationcited by 26
  35. 2024AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuningcited by 25
  36. 2019The Value of Collaboration in Convex Machine Learning with Differential Privacycited by 24
  37. 2024CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimizationcited by 14
  38. 2024LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Modelscited by 12

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 5.

12 papers.

  1. 2025Decentralized Low-Rank Fine-Tuning of Large Language Modelscited by 12
  2. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  3. 2025FLoRIST: Singular Value Thresholding for Efficient and Accurate Federated Fine-Tuning of Large Language Modelscited by 4
  4. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 4
  5. 2025pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Modelscited by 3
  6. 2024Leveraging Large Language Models and Topic Modeling for Toxicity Classificationcited by 2
  7. 2024Revisiting Federated Fine-Tuning: A Single Communication Round is Enough for Foundation Modelscited by 1
  8. 2026Leveraging Soft Prompts for Privacy Attacks in Federated Prompt Tuningcited by 1
  9. 2026REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotationscited by 1
  10. 2026SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Rankscited by 1
  11. 2025FLAME: Towards Federated Fine-Tuning Large Language Models Through Adaptive SMoEcited by 0
  12. 2026FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantizationcited by 0