# DEeR: Deviation Eliminating and Noise Regulating for Privacy-preserving Federated Low-rank Adaptation

arXiv: https://arxiv.org/abs/2410.12926 · HTML: https://mlsanity.com/paper/2410.12926/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 65 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Builds on (most cited first, 30 of 39)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Kvasir-SEG: A Segmented Polyp Dataset](https://mlsanity.com/paper/1911.07069/) (2019)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [PraNet: Parallel Reverse Attention Network for Polyp Segmentation](https://mlsanity.com/paper/2006.11392/) (2020)
- [Opacus: User-Friendly Differential Privacy Library in PyTorch](https://mlsanity.com/paper/2109.12298/) (2021)
- [VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks](https://mlsanity.com/paper/2112.06825/) (2021)
- [Side Adapter Network for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2302.12242/) (2023)
- [Mitigating the Alignment Tax of RLHF](https://mlsanity.com/paper/2309.06256/) (2023)
- [Mixture of LoRA Experts](https://mlsanity.com/paper/2404.13628/) (2024)
- [Improving LoRA in Privacy-preserving Federated Learning](https://mlsanity.com/paper/2403.12313/) (2024)
- [Non-convex Optimization for Machine Learning](https://mlsanity.com/paper/1712.07897/) (2017)
- [SAM-Med2D](https://mlsanity.com/paper/2308.16184/) (2023)
- [Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models](https://mlsanity.com/paper/2401.06432/) (2024)
- [When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications](https://mlsanity.com/paper/2310.18339/) (2023)
- [SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models](https://mlsanity.com/paper/2308.06522/) (2023)
- [When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods](https://mlsanity.com/paper/2212.10025/) (2022)
- [Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Model](https://mlsanity.com/paper/2401.17868/) (2024)
- [Large AI Models in Health Informatics: Applications, Challenges, and the Future](https://mlsanity.com/paper/2303.11568/) (2023)
- [LoRA Dropout as a Sparsity Regularizer for Overfitting Control](https://mlsanity.com/paper/2404.09610/) (2024)

## Built on by (most cited first, 4 of 4)

- [FedBM: Stealing Knowledge from Pre-trained Language Models for Heterogeneous Federated Learning](https://mlsanity.com/paper/2502.16832/) (2025)
- [DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models](https://mlsanity.com/paper/2505.21382/) (2025)
- [FedLoDrop: Federated LoRA with Dropout for Generalized LLM Fine-tuning](https://mlsanity.com/paper/2510.12078/) (2025)
- [Personalized Federated Learning with Residual Fisher Information for Medical Image Segmentation](https://mlsanity.com/paper/2603.14848/) (2026)

## Abstract (arXiv)

Integrating low-rank adaptation (LoRA) with federated learning (FL) has received widespread attention recently, aiming to adapt pretrained foundation models (FMs) to downstream medical tasks via privacy-preserving decentralized training. However, owing to the direct combination of LoRA and FL, current methods generally undergo two problems, i.e., aggregation deviation, and differential privacy (DP) noise amplification effect. To address these problems, we propose a novel privacy-preserving federated finetuning framework called underlineDeviation underlineEliminating and Noisunderlinee underlineRegulating (DEeR). Specifically, we firstly theoretically prove that the necessary condition to eliminate aggregation deviation is guaranteing the equivalence between LoRA parameters of clients. Based on the theoretical insight, a deviation eliminator is designed to utilize alternating minimization algorithm to iteratively optimize the zero-initialized and non-zero-initialized parameter matrices of LoRA, ensuring that aggregation deviation always be zeros during training. Furthermore, we also conduct an in-depth analysis of the noise amplification effect and find that this problem is mainly caused by the ``linear relationship'' between DP noise and LoRA parameters. To suppress the noise amplification effect, we propose a noise regulator that exploits two regulator factors to decouple relationship between DP and LoRA, thereby achieving robust privacy protection and excellent finetuning performance. Additionally, we perform comprehensive ablated experiments to verify the effectiveness of the deviation eliminator and noise regulator. DEeR shows better performance on public medical datasets in comparison with state-of-the-art approaches. The code is available at https://github.com/CUHK-AIM-Group/DEeR.