mlsanityData to 7 Oct 2026

Towards More Realistic Extraction Attacks: An Adversarial Perspective

arXiv 2407.02596 ↗First posted 2 Jul 2024cs.CR cs.CL cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Language models are prone to memorizing their training data, making them vulnerable to extraction attacks. While existing research often examines isolated setups, such as a single model or a fixed prompt, real-world adversaries have a considerably larger attack surface due to access to models across various sizes and checkpoints, and repeated prompting. In this paper, we revisit extraction attacks from an adversarial perspective -- with multi-faceted access to the underlying data. We find significant churn in extraction trends, i.e., even unintuitive changes to the prompt, or targeting smaller models and earlier checkpoints, can extract distinct information. By combining multiple attacks, our adversary doubles (2 ×) the extraction risks, persisting even under mitigation strategies like data deduplication. We conclude with four case studies, including detecting pre-training data, copyright violations, extracting personally identifiable information, and attacking closed-source models, showing how our more realistic adversary can outperform existing adversaries in the literature.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 53 of its 70 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

53 of 70 references.

References · 53

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2023A Survey of Large Language Modelscited by 2,537
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  9. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  10. 2021Membership Inference Attacks From First Principlescited by 839
  11. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  12. 2023Large Language Modelscited by 774
38 more
  1. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  2. 2023Extracting Training Data from Diffusion Modelscited by 723
  3. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  4. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  5. 2023The Falcon Series of Open Language Modelscited by 526
  6. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  7. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  8. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  9. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  10. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  11. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  12. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  13. 2023How is ChatGPT's behavior changing over time?cited by 265
  14. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  15. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  16. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  17. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  18. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  19. 2018Large scale distributed neural network training through online distillationcited by 144
  20. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  21. 2023Copyright Violations and Large Language Modelscited by 108
  22. 2024LLM Dataset Inference: Did you train on my dataset?cited by 107
  23. 2018Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goalscited by 103
  24. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 97
  25. 2024The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performancecited by 97
  26. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 91
  27. 2023GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformercited by 84
  28. 2022Training Trajectories of Language Models Across Scalescited by 80
  29. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
  30. 2021Probing Across Time: What Does RoBERTa Know and When?cited by 73
  31. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 72
  32. 2023Bag of Tricks for Training Data Extraction from Language Modelscited by 67
  33. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 54
  34. 2024Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluationcited by 37
  35. 2024Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMscited by 33
  36. 2024An Early Categorization of Prompt Injection Attacks on Large Language Modelscited by 28
  37. 2023Selective Pre-training for Private Fine-tuningcited by 22
  38. 2023Model Leeching: An Extraction Attack Targeting LLMscited by 22

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 1.

8 papers.

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  4. 2025Exact Unlearning of Finetuning Data via Model Merging at Scalecited by 14
  5. 2025LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasurescited by 8
  6. 2024PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMscited by 6
  7. 2024MemHunter: Automated and Verifiable Memorization Detection at Dataset-scale in LLMscited by 2
  8. 2026AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspectivecited by 1