mlsanityData to 14 Oct 2026

Privacy-Preserving Parameter-Efficient Fine-Tuning for Large Language Model Services

arXiv 2305.06212 ↗First posted 10 May 2023cs.CL cs.CRcited by 43

Not on the map. Why?

Abstract (arXiv)

Parameter-Efficient Fine-Tuning (PEFT) provides a practical way for users to customize Large Language Models (LLMs) with their private data in LLM service scenarios. However, the inherently sensitive nature of private data demands robust privacy preservation measures during the customization of LLM services to ensure data security, maintain user trust, and comply with stringent regulatory standards. Based on PEFT, we propose Privacy-Preserving Parameter-Efficient Fine-Tuning (RAPT), a framework that offers privacy protection for LLM services. RAPT adopts a local privacy approach, enabling users to privatize their data locally using a text-to-text local differential privacy mechanism. Since PEFT performs poorly when directly trained on privatized data, we introduce a novel privatized token reconstruction task that is trained jointly with the downstream task, allowing LLMs to learn better task-dependent representations. Despite the simplicity of our framework, experiments show that RAPT achieves competitive performance across tasks while providing privacy guarantees against adversaries.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 32 of its 67 references that are in the map. First posted on arXiv 10 May 2023.

Builds on

32 of 67 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  9. 2020Extracting Training Data from Large Language Modelscited by 2,189
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  11. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  12. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
20 more
  1. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  2. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  3. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 285
  4. 2020Information Leakage in Embedding Modelscited by 203
  5. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  6. 2019Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbationscited by 119
  7. 2021Large-Scale Differentially Private BERTcited by 118
  8. 2022Should You Mask 15% in Masked Language Modeling?cited by 118
  9. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 88
  10. 2020Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairnesscited by 67
  11. 2021Selective Differential Privacy for Language Modelingcited by 62
  12. 2021Natural Language Understanding with Privacy-Preserving BERTcited by 55
  13. 2018Privacy-preserving Neural Representations of Textcited by 54
  14. 2020Differentially Private Language Models Benefit from Public Pre-trainingcited by 47
  15. 2020Emergence of Separable Manifolds in Deep Language Representationscited by 40
  16. 2020TextHide: Tackling Data Privacy in Language Understanding Taskscited by 33
  17. 2022Just Fine-tune Twice: Selective Differential Privacy for Large Language Modelscited by 32
  18. 2021CAPE: Context-Aware Private Embeddings for Private Language Learningcited by 22
  19. 2022On Robust Prefix-Tuning for Text Classificationcited by 15
  20. 2023Attack Named Entity Recognition by Entity Boundary Interferencecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 19; 2025: 9; 2026: 1.

43 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  2. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 189
  3. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 142
  4. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  5. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 82
  6. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 79
  7. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
  8. 2024Building Guardrails for Large Language Modelscited by 65
  9. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 51
  10. 2023Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learningcited by 41
  11. 2023PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Modelscited by 37
  12. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
31 more
  1. 2023Split-and-Denoise: Protect large language model inference with local differential privacycited by 32
  2. 2023Interactive Natural Language Processingcited by 31
  3. 2023Integration of Large Language Models and Federated Learningcited by 31
  4. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  5. 2024PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restorationcited by 28
  6. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  7. 2024Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuningcited by 24
  8. 2024IncogniText: Privacy-enhancing Conditional Text Anonymization via LLM-based Private Attribute Randomizationcited by 23
  9. 2025Data Poisoning in Deep Learning: A Surveycited by 20
  10. 2023PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learnerscited by 14
  11. 2024Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directionscited by 14
  12. 2024No Free Lunch Theorem for Privacy-Preserving LLM Inferencecited by 13
  13. 2023A Split-and-Privatize Framework for Large Language Model Fine-Tuningcited by 11
  14. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  15. 2024Risks, Causes, and Mitigations of Widespread Deployments of Large Language Models (LLMs): A Surveycited by 8
  16. 2024Imagining a Future of Designing with AI: Dynamic Grounding, Constructive Negotiation, and Sustainable Motivationcited by 7
  17. 2024Privacy-preserving Fine-tuning of Large Language Models through Flatnesscited by 6
  18. 2025LLM Access Shield: Domain-Specific LLM Framework for Privacy Policy Compliancecited by 6
  19. 2024Locally Differentially Private In-Context Learningcited by 5
  20. 2024The Tug of War Within: Mitigating the Fairness-Privacy Conflicts in Large Language Modelscited by 5
  21. 2025Privacy-Preserving LLM Embedding Transmission for End-Cloud Collaborationcited by 5
  22. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  23. 2025Can Differentially Private Fine-tuning LLMs Protect Against Privacy Attacks?cited by 4
  24. 2024PFID: Privacy First Inference Delegation Framework for LLMscited by 3
  25. 2024Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Datacited by 2
  26. 2024AI Delegates with a Dual Focus: Ensuring Privacy and Strategic Self-Disclosurecited by 2
  27. 2025DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learningcited by 2
  28. 2025Learning Obfuscations Of LLM Embedding Sequences: Stained Glass Transformcited by 2
  29. 2025Semantic Encryption: Secure and Effective Interaction with Cloud-based Large Language Models via Semantic Transformationcited by 0
  30. 2025AMAQ: Adaptive Mixed-bit Activation Quantization for Collaborative Parameter Efficient Fine-tuningcited by 0
  31. 2026Continual Pretraining on Encrypted Synthetic Data for Privacy-Preserving LLMscited by 0