mlsanityData to 7 Oct 2026

Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs

arXiv 2403.04801 ↗First posted 5 Mar 2024cs.CLcited by 33

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

In this paper, we introduce a black-box prompt optimization method that uses an attacker LLM agent to uncover higher levels of memorization in a victim agent, compared to what is revealed by prompting the target model with the training data directly, which is the dominant approach of quantifying memorization in LLMs. We use an iterative rejection-sampling optimization process to find instruction-based prompts with two main characteristics: (1) minimal overlap with the training data to avoid presenting the solution directly to the model, and (2) maximal overlap between the victim model's output and the training data, aiming to induce the victim to spit out training data. We observe that our instruction-based prompts generate outputs with 23.7% higher overlap with training data compared to the baseline prefix-suffix measurements. Our findings show that (1) instruction-tuned models can expose pre-training data as much as their base-models, if not more so, (2) contexts other than the original training data can lead to leakage, and (3) using instructions proposed by other LLMs can open a new avenue of automated attacks that we should further study and explore. The code can be found at https://github.com/Alymostafa/Instruction_based_attack .

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 38 of its 55 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

38 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan

Other references · 37

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2024Mixtral of Expertscited by 1,785
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
25 more
  1. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  2. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  3. 2023Extracting Training Data from Diffusion Modelscited by 723
  4. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  5. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  6. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  7. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  8. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  9. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  10. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  11. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  12. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  13. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 366
  14. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  15. 2023The False Promise of Imitating Proprietary LLMscited by 254
  16. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  17. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  18. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  19. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  20. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  21. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 97
  22. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  23. 2023Bag of Tricks for Training Data Extraction from Language Modelscited by 67
  24. 2023Make Them Spill the Beans! Coercive Knowledge Extraction from (Production) LLMscited by 26
  25. 2023Are Chatbots Ready for Privacy-Sensitive Applications? An Investigation into Input Regurgitation and Prompt-Induced Sanitizationcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 13; 2026: 7.

33 papers.

Sort:

By area:

  1. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  2. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  3. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 55
  4. 2024Measuring memorization in language models via probabilistic extractioncited by 40
  5. 2024CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generationcited by 35
  6. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  7. 2024Manipulation Facing Threats: Evaluating Physical Vulnerabilities in End-to-End Vision Language Action Modelscited by 16
  8. 2024Measuring memorization in RLHF for code completioncited by 12
  9. 2025Blackbox Model Provenance via Palimpsestic Membership Inferencecited by 11
  10. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 9
  11. 2024Towards More Realistic Extraction Attacks: An Adversarial Perspectivecited by 8
  12. 2025LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasurescited by 8
21 more
  1. 2025Memorize or Generalize? Evaluating LLM Code Generation with Code Rewritingcited by 6
  2. 2026Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Modelscited by 6
  3. 2024Prompt Recovery for Image Generation Models: A Comparative Study of Discrete Optimizerscited by 5
  4. 2025Labeling Free-text Data using Language Model Ensemblescited by 5
  5. 2024Uncovering Latent Memories: Assessing Data Leakage and Memorization Patterns in Frontier AI Modelscited by 4
  6. 2024Sequence-Level Leakage Risk of Training Data in Large Language Modelscited by 4
  7. 2025AdaToken-3D: Dynamic Spatial Gating for Efficient 3D Large Multimodal-Models Reasoningcited by 4
  8. 2025What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requestscited by 4
  9. 2025Leaner Training, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRAcited by 3
  10. 2025Retracing the Past: LLMs Emit Training Data When They Get Lostcited by 3
  11. 2024MemHunter: Automated and Verifiable Memorization Detection at Dataset-scale in LLMscited by 2
  12. 2024On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMscited by 2
  13. 2026C^2ROPE: Causal Continuous Rotary Positional Encoding for 3D Large Multimodal-Models Reasoningcited by 2
  14. 2026GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuningcited by 2
  15. 2025Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Modelscited by 1
  16. 2026Beyond Indistinguishability: Measuring Extraction Risk in LLM APIscited by 1
  17. 2026LLMs Can Leak Training Data But Do They Want To? A Propensity-Aware Evaluation of Memorization in LLMscited by 1
  18. 2025Dual Refinement Cycle Learning: Unsupervised Text Classification of Mamba and Community Detection on Text Attributed Graphcited by 0
  19. 2025CTIGuardian: A Few-Shot Framework for Mitigating Privacy Leakage in Fine-Tuned LLMscited by 0
  20. 2026Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablationcited by 0
  21. 2026Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judge
  2. 2024Rethinking LLM Memorization through the Lens of Adversarial Compression
  3. 2026From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents
  4. 2025Memory Injection Attacks on LLM Agents via Query-Only Interaction
  5. 2025Learn to Memorize: Optimizing LLM-based Agents with Adaptive Memory Framework
  6. 2026From Attack Success to Attack Severity: Counterfactual Memory Attacks on LLM Agents
  7. 2026Automated Membership Inference Attacks (AutoMIA): Discovering MIA Signal Computations using LLM Agents
  8. 2026The Price of Safety: Benign-Case Utility and Token Overhead of Memory-Poisoning Defenses in LLM Agents