mlsanityData to 14 Oct 2026

Fine-Tuning Language Models with Differential Privacy through Adaptive Noise Allocation

arXiv 2410.02912 ↗First posted 3 Oct 2024cs.AI cs.CL cs.CRcited by 14

Not on the map. Why?

Abstract (arXiv)

Language models are capable of memorizing detailed patterns and information, leading to a double-edged effect: they achieve impressive modeling performance on downstream tasks with the stored knowledge but also raise significant privacy concerns. Traditional differential privacy based training approaches offer robust safeguards by employing a uniform noise distribution across all parameters. However, this overlooks the distinct sensitivities and contributions of individual parameters in privacy protection and often results in suboptimal models. To address these limitations, we propose ANADP, a novel algorithm that adaptively allocates additive noise based on the importance of model parameters. We demonstrate that ANADP narrows the performance gap between regular fine-tuning and traditional DP fine-tuning on a series of datasets while maintaining the required privacy constraints.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 20 of its 30 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

20 of 30 references.

References · 20

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2016Deep Learning with Differential Privacycited by 2,728
  10. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
8 more
  1. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  2. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  3. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  4. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 230
  5. 2023Privacy Auditing with One (1) Training Runcited by 123
  6. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 88
  7. 2023Privacy Issues in Large Language Models: A Surveycited by 60
  8. 2020Differentially Private Language Models Benefit from Public Pre-trainingcited by 47

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 5.

14 papers.

Sort:

By area:

  1. 2025Parameter-Efficient Fine-Tuning with Differential Privacy for Robust Instruction Adaptation in Large Language Modelscited by 5
  2. 2025Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Modelscited by 4
  3. 2025A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challengescited by 3
  4. 2026Differentially Private Subspace Fine-Tuning for Large Language Modelscited by 2
  5. 2025Inclusive Federated Learning Through Compliance-Weighted Noise Allocation in Healthcare AIcited by 1
  6. 2025SecP-Tuning: Efficient Privacy-Preserving Prompt Tuning for Large Language Models via MPCcited by 1
  7. 2025Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Trainingcited by 1
  8. 2025Towards the Anonymization of the Language Modelingcited by 0
  9. 2025PPFL-RDSN: Privacy-Preserving Federated Learning-based Residual Dense Spatial Networks for Encrypted Lossy Image Reconstructioncited by 0
  10. 2025Perturb Your Data: Paraphrase-Guided Training Data Watermarkingcited by 0
  11. 2026PrivMedChat: End-to-End Differentially Private RLHF for Medical Dialogue Systemscited by 0
  12. 2026When Privacy Hurts Mergeability: Geometry-Aware Model Merging under Differential Privacycited by 0
2 more
  1. 2026PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatizationcited by 0
  2. 2026The Privacy-Hallucination Tradeoff in Differentially Private Language Modelscited by 0