mlsanityData to 14 Oct 2026

Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning

arXiv 2305.11759 ↗First posted 19 May 2023cs.CL cs.AIcited by 40

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are known to memorize significant portions of their training data. Parts of this memorized content have been shown to be extractable by simply querying the model, which poses a privacy risk. We present a novel approach which uses prompt-tuning to control the extraction rates of memorized content in LLMs. We present two prompt training strategies to increase and decrease extraction rates, which correspond to an attack and a defense, respectively. We demonstrate the effectiveness of our techniques by using models from the GPT-Neo family on a public benchmark. For the 1.3B parameter GPT-Neo model, our attack yields a 9.3 percentage point increase in extraction rate compared to our baseline. Our defense can be tuned to achieve different privacy-utility trade-offs by a user-specified hyperparameter. We achieve an extraction rate reduction of up to 97.7% relative to our baseline, with a perplexity increase of 16.9%.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

25 of 36 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2016Deep Learning with Differential Privacycited by 2,728
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
13 more
  1. 2020Extracting Training Data from Large Language Modelscited by 2,189
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  3. 2019How Can We Know What Language Models Know?cited by 959
  4. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  5. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  6. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  7. 2021Counterfactual Memorization in Neural Language Modelscited by 156
  8. 2022Federated Unlearning: How to Efficiently Erase a Client in FL?cited by 95
  9. 2020Defending against Backdoors in Federated Learning with Robust Learning Ratecited by 82
  10. 2022AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Modelcited by 57
  11. 2022RelaxLoss: Defending Membership Inference Attacks without Losing Utilitycited by 37
  12. 2022Differentially Private Decoding in Large Language Modelscited by 34
  13. 2021An Efficient DP-SGD Mechanism for Large Scale NLP Modelscited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 22; 2025: 8; 2026: 3.

40 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  2. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  3. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 131
  4. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  5. 2023Copyright Violations and Large Language Modelscited by 108
  6. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  7. 2024Safeguarding Large Language Models: A Surveycited by 78
  8. 2024Building Guardrails for Large Language Modelscited by 65
  9. 2023On the steerability of large language models toward data-driven personascited by 53
  10. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 51
  11. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  12. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 37
28 more
  1. 2023The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Riskscited by 36
  2. 2023Integration of Large Language Models and Federated Learningcited by 31
  3. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  4. 2024PII-Compass: Guiding LLM training data extraction prompts towards the target PII via groundingcited by 24
  5. 2024Unlocking Memorization in Large Language Models with Dynamic Soft Promptingcited by 21
  6. 2024Ethos: Rectifying Language Models in Orthogonal Parameter Spacecited by 20
  7. 2023Quantifying and Analyzing Entity-level Memorization in Large Language Modelscited by 18
  8. 2024Special Characters Attack: Toward Scalable Training Data Extraction From Large Language Modelscited by 17
  9. 2024MIA-Tuner: Adapting Large Language Models as Pre-training Text Detectorcited by 15
  10. 2024Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directionscited by 14
  11. 2024Do LLMs Know to Respect Copyright Notice?cited by 13
  12. 2024A Comprehensive Survey of Contamination Detection Methods in Large Language Modelscited by 12
  13. 2025SALAD: Systematic Assessment of Machine Unlearning on LLM-Aided Hardware Designcited by 12
  14. 2025VeriLeaky: Navigating IP Protection vs Utility in Fine-Tuning for LLM-Driven Verilog Codingcited by 11
  15. 2025Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Trainingcited by 10
  16. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 10
  17. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  18. 2025Mitigating Memorization in LLMs using Activation Steeringcited by 8
  19. 2024PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMscited by 7
  20. 2024Enhancing Data Privacy in Large Language Models through Private Association Editingcited by 6
  21. 2024Forecasting Credit Ratings: A Case Study where Traditional Methods Outperform Generative LLMscited by 5
  22. 2024Large Language Models Memorize Sensor Datasets! Implications on Human Activity Recognition Researchcited by 3
  23. 2024Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Datacited by 2
  24. 2025A Lightweight Method to Disrupt Memorized Sequences in LLMcited by 2
  25. 2026Data-Free Privacy-Preserving for LLMs via Model Inversion and Selective Unlearningcited by 1
  26. 2026Orchestration-Free Customer Service Automation: A Privacy-Preserving and Flowchart-Guided Frameworkcited by 1
  27. 2025Malicious and Unintentional Disclosure Risks in Large Language Models for Code Generationcited by 0
  28. 2026Cascade: Hierarchical Recoverability Control for Large Language Model Unlearningcited by 0