mlsanityData to 7 Oct 2026

DEeR: Deviation Eliminating and Noise Regulating for Privacy-preserving Federated Low-rank Adaptation

arXiv 2410.12926 ↗First posted 16 Oct 2024cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Integrating low-rank adaptation (LoRA) with federated learning (FL) has received widespread attention recently, aiming to adapt pretrained foundation models (FMs) to downstream medical tasks via privacy-preserving decentralized training. However, owing to the direct combination of LoRA and FL, current methods generally undergo two problems, i.e., aggregation deviation, and differential privacy (DP) noise amplification effect. To address these problems, we propose a novel privacy-preserving federated finetuning framework called underlineDeviation underlineEliminating and Noisunderlinee underlineRegulating (DEeR). Specifically, we firstly theoretically prove that the necessary condition to eliminate aggregation deviation is guaranteing the equivalence between LoRA parameters of clients. Based on the theoretical insight, a deviation eliminator is designed to utilize alternating minimization algorithm to iteratively optimize the zero-initialized and non-zero-initialized parameter matrices of LoRA, ensuring that aggregation deviation always be zeros during training. Furthermore, we also conduct an in-depth analysis of the noise amplification effect and find that this problem is mainly caused by the ``linear relationship'' between DP noise and LoRA parameters. To suppress the noise amplification effect, we propose a noise regulator that exploits two regulator factors to decouple relationship between DP and LoRA, thereby achieving robust privacy protection and excellent finetuning performance. Additionally, we perform comprehensive ablated experiments to verify the effectiveness of the deviation eliminator and noise regulator. DEeR shows better performance on public medical datasets in comparison with state-of-the-art approaches. The code is available at https://github.com/CUHK-AIM-Group/DEeR.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 39 of its 65 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

39 of 65 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Segment Anythingcited by 8,158
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2019Advances and Open Problems in Federated Learningcited by 3,021
  8. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  9. 2016Deep Learning with Differential Privacycited by 2,682
  10. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  11. 2022Visual Prompt Tuningcited by 1,674
  12. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
27 more
  1. 2019Kvasir-SEG: A Segmented Polyp Datasetcited by 588
  2. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  3. 2020PraNet: Parallel Reverse Attention Network for Polyp Segmentationcited by 460
  4. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  5. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  6. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  7. 2023Mitigating the Alignment Tax of RLHFcited by 181
  8. 2024Mixture of LoRA Expertscited by 179
  9. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  10. 2017Non-convex Optimization for Machine Learningcited by 132
  11. 2023SAM-Med2Dcited by 123
  12. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 114
  13. 2023When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applicationscited by 110
  14. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  15. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  16. 2024Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Modelcited by 67
  17. 2023Large AI Models in Health Informatics: Applications, Challenges, and the Futurecited by 61
  18. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  19. 2023Make Landscape Flatter in Differentially Private Federated Learningcited by 33
  20. 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
  21. 2024Federated LoRA with Sparse Communicationcited by 26
  22. 2021Personalized Retrogress-Resilient Framework for Real-World Medical Federated Learningcited by 25
  23. 2023Low-Parameter Federated Learning with Large Language Modelscited by 22
  24. 2024FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learningcited by 16
  25. 2024DP-DyLoRA: Fine-Tuning Transformer-Based Models On-Device under Differentially Private Federated Learning using Dynamic Low-Rank Adaptationcited by 9
  26. 2024Communication-Efficient Personalized Federated Learning for Speech-to-Text Taskscited by 8
  27. 2024SPD-CFL: Stepwise Parameter Dropout for Efficient Continual Federated Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025FedBM: Stealing Knowledge from Pre-trained Language Models for Heterogeneous Federated Learningcited by 2
  2. 2025DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Modelscited by 2
  3. 2025FedLoDrop: Federated LoRA with Dropout for Generalized LLM Fine-tuningcited by 2
  4. 2026Personalized Federated Learning with Residual Fisher Information for Medical Image Segmentationcited by 0