mlsanityData to 14 Oct 2026

DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models

arXiv 2605.10272 ↗First posted 11 May 2026cs.LG cs.AI cs.CRcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) enables the collaborative training of large-scale language models (LLMs) across edge devices while keeping user data on-device. However, FL still exposes sensitive information through client-provided gradients. Differentially private stochastic gradient descent (DP-SGD) mitigates this risk by clipping each client's contribution to a threshold C and adding noise proportional to C. Existing adaptive clipping techniques dynamically adjust C but demand tedious hyperparameter tuning, which can erode the privacy budget. In this paper, we introduce DP-LAC, a method that first estimates an initial clipping threshold within an order of magnitude of the optimum using private histogram estimation, and then adapts this threshold during training without consuming additional privacy budget or introducing new hyperparameters. Empirical results show that DP-LAC outperforms both state-of-the-art adaptive clipping methods and vanilla DP-SGD, achieving an average accuracy gain of 6.6%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 36 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

30 of 36 references.

References · 30

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2019Advances and Open Problems in Federated Learningcited by 3,046
  7. 2016Deep Learning with Differential Privacycited by 2,728
  8. 2020Extracting Training Data from Large Language Modelscited by 2,189
  9. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  10. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  11. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  12. 2017Renyi Differential Privacycited by 650
18 more
  1. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  2. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 459
  3. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  4. 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 242
  5. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 236
  6. 2019Differentially Private Learning with Adaptive Clippingcited by 220
  7. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  8. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 133
  9. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 116
  10. 2021Hyperparameter Tuning with Renyi Differential Privacycited by 104
  11. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 88
  12. 2019Hypothesis Testing Interpretations and Renyi Differential Privacycited by 85
  13. 2018Private Selection from Private Candidatescited by 75
  14. 2021Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacycited by 69
  15. 2021Dynamic Differential-Privacy Preserving SGDcited by 30
  16. 2024DP-DyLoRA: Fine-Tuning Transformer-Based Models On-Device under Differentially Private Federated Learning using Dynamic Low-Rank Adaptationcited by 9
  17. 2025Towards hyperparameter-free optimization with differential privacycited by 7
  18. 2022Algorithms for bounding contribution for histogram estimation under user-level privacycited by 4

Built on by

Not cited yet.