mlsanityData to 7 Oct 2026

Memorization in NLP Fine-tuning Methods

arXiv 2205.12506 ↗First posted 25 May 2022cs.CL cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

Large language models are shown to present privacy risks through memorization of training data, and several recent works have studied such risks for the pre-training phase. Little attention, however, has been given to the fine-tuning phase and it is not well understood how different fine-tuning methods (such as fine-tuning the full model, the model head, and adapter) compare in terms of memorization risk. This presents increasing concern as the "pre-train and fine-tune" paradigm proliferates. In this paper, we empirically study memorization of fine-tuning methods using membership inference and extraction attacks, and show that their susceptibility to attacks is very different. We observe that fine-tuning the head of the model has the highest susceptibility to attacks, whereas fine-tuning smaller adapters appears to be less vulnerable to known extraction attacks.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 32 of its 37 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

32 of 37 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  12. 2020Extracting Training Data from Large Language Modelscited by 2,144
20 more
  1. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  2. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  3. 2021Membership Inference Attacks From First Principlescited by 839
  4. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  5. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  6. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  7. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  8. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  9. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
  10. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  11. 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 142
  12. 2019Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learningcited by 117
  13. 2020Revisiting Membership Inference Under Realistic Assumptionscited by 108
  14. 2019Gmail Smart Compose: Real-Time Assisted Writingcited by 97
  15. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 91
  16. 2019Analyzing Information Leakage of Updates to Natural Language Modelscited by 89
  17. 2020Privacy in Deep Learning: A Surveycited by 70
  18. 2021Memorisation versus Generalisation in Pre-trained Language Modelscited by 32
  19. 2021Benchmarking Differential Privacy and Federated Learning for BERT Modelscited by 28
  20. 2020KART: Parameterization of Privacy Leakage Scenarios from Pre-trained Language Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 12; 2024: 16; 2025: 8; 2026: 6.

43 papers.

Sort:

By area:

  1. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  2. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  3. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)cited by 159
  4. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 80
  5. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  6. 2024Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attentioncited by 57
  7. 2025How much do language models memorize?cited by 53
  8. 2023Exploring Memorization in Fine-tuned Language Modelscited by 44
  9. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 38
  10. 2024CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generationcited by 35
  11. 2024On the Privacy Risk of In-context Learningcited by 31
  12. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
31 more
  1. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  2. 2024Security of AI Agentscited by 24
  3. 2024KBLaM: Knowledge Base augmented Language Modelcited by 16
  4. 2024DP-TabICL: In-Context Learning with Differentially Private Tabular Datacited by 11
  5. 2025Privacy-Aware RAG: Secure and Isolated Knowledge Retrievalcited by 10
  6. 2023Does fine-tuning GPT-3 with the OpenAI API leak personally-identifiable information?cited by 9
  7. 2023Gradient-Free Privacy Leakage in Federated Language Models through Selective Weight Tamperingcited by 9
  8. 2025LoRA-Leak: Membership Inference Attacks Against LoRA Fine-tuned Language Modelscited by 7
  9. 2023Matching Pairs: Attributing Fine-Tuned Models to their Pre-Trained Large Language Modelscited by 6
  10. 2024Privacy-preserving Fine-tuning of Large Language Models through Flatnesscited by 6
  11. 2024Copyright-Protected Language Generation via Adaptive Model Fusioncited by 6
  12. 2024Locally Differentially Private In-Context Learningcited by 5
  13. 2025Enhancing Privacy-Utility Trade-offs to Mitigate Memorization in Diffusion Modelscited by 5
  14. 2026Window-based Membership Inference Attacks Against Fine-tuned Large Language Modelscited by 5
  15. 2023Trade-Offs Between Fairness and Privacy in Language Modelingcited by 4
  16. 2023Memorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translationcited by 4
  17. 2024Strong Copyright Protection for Language Models via Adaptive Model Fusioncited by 4
  18. 2026Unveiling Privacy Risks in Multi-modal Large Language Models: Task-specific Vulnerabilities and Mitigation Challengescited by 4
  19. 2024Studying and Recommending Information Highlighting in Stack Overflow Answerscited by 2
  20. 2024LoRA Provides Differential Privacy by Design via Random Sketchingcited by 2
  21. 2025Memory-Efficient Differentially Private Training with Gradient Random Projectioncited by 2
  22. 2026Private Seeds, Public LLMs: Realistic and Privacy-Preserving Synthetic Data Generationcited by 2
  23. 2022Swing Distillation: A Privacy-Preserving Knowledge Distillation Frameworkcited by 1
  24. 2025A Lightweight Method to Disrupt Memorized Sequences in LLMcited by 1
  25. 2026Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMscited by 1
  26. 2026Vision Language Model Helps Private Information De-Identification in Vision Datacited by 1
  27. 2023Improving Reliability of Fine-tuning with Block-wise Optimisationcited by 0
  28. 2024Inside the Black Box: Detecting Data Leakage in Pre-trained Language Encoderscited by 0
  29. 2025On the Performance of Differentially Private Optimization with Heavy-Tail Class Imbalancecited by 0
  30. 2025Early Detection and Reduction of Memorisation for Domain Adaptation and Instruction Tuningcited by 0
  31. 2026Clinically Grounded Privacy Evaluation of Medical LMscited by 0