mlsanityData to 14 Oct 2026

Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMs

arXiv 2506.17353 ↗First posted 20 Jun 2025cs.CR cs.AIcited by 6

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

The increasing demand for domain-specific and human-aligned Large Language Models (LLMs) has led to the widespread adoption of Supervised Fine-Tuning (SFT) techniques. SFT datasets often comprise valuable instruction-response pairs, making them highly valuable targets for potential extraction. This paper studies this critical research problem for the first time. We start by formally defining and formulating the problem, then explore various attack goals, types, and variants based on the unique properties of SFT data in real-world scenarios. Based on our analysis of extraction behaviors of direct extraction, we develop a novel extraction method specifically designed for SFT models, called Differentiated Data Extraction (DDE), which exploits the confidence levels of fine-tuned models and their behavioral differences from pre-trained base models. Through extensive experiments across multiple domains and scenarios, we demonstrate the feasibility of SFT data extraction using DDE. Our results show that DDE consistently outperforms existing extraction baselines in all attack settings. To counter this new attack, we propose a defense mechanism that mitigates DDE attacks with minimal impact on model performance. Overall, our research reveals hidden data leak risks in fine-tuned LLMs and provides insights for developing more secure models.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 93 references that are in the map. First posted on arXiv 20 Jun 2025.

Builds on

53 of 93 references.

References · 53

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
38 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  4. 2020Extracting Training Data from Large Language Modelscited by 2,189
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  9. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  12. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  13. 2023LIMA: Less Is More for Alignmentcited by 1,100
  14. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  15. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  16. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  17. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 678
  18. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  19. 2016Stealing Machine Learning Models via Prediction APIscited by 619
  20. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  21. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  22. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  23. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  24. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 156
  25. 2019Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learningcited by 122
  26. 2018Stealing Hyperparameters in Machine Learningcited by 118
  27. 2019Analyzing Information Leakage of Updates to Natural Language Modelscited by 90
  28. 2024A Closer Look at the Limitations of Instruction Tuningcited by 77
  29. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 74
  30. 2021Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecturecited by 67
  31. 2022Membership Inference Attacks and Defenses in Neural Network Pruningcited by 45
  32. 2024CYCLE: Learning to Self-Refine the Code Generationcited by 42
  33. 2024Meta Large Language Model Compiler: Foundation Models of Compiler Optimizationcited by 39
  34. 2024How Beginning Programmers and Code LLMs (Mis)read Each Othercited by 31
  35. 2023Traces of Memorisation in Large Language Models for Codecited by 29
  36. 2024Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Functioncited by 24
  37. 2022Membership Inference Attacks and Generalization: A Causal Perspectivecited by 17
  38. 2024Exploring Multi-Lingual Bias of Large Code Models in Code Generationcited by 17

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026On Protecting Agentic Systems' Intellectual Property via Watermarkingcited by 8
  2. 2026Echoes within the Reasoning: Stealthy and Effective Watermarking via Chain of Thoughtcited by 2
  3. 2026Functional Subspace Watermarking for Large Language Modelscited by 0
  4. 2026Reconstruction of Personally Identifiable Information from Proprietary Data in Supervised Fine-Tuned Modelscited by 0
  5. 2026Feyospace-v1: How the Cyber Mercury Seven Trained Frontier Cyber Modelscited by 0
  6. 2026The Privacy Fallacy of Crowdsourced Fine-Tuning: Extracting Proprietary Data via Topic-Based Poisoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!
  2. 2025A Survey on Model Extraction Attacks and Defenses for Large Language Models
  3. 2025How Much Do Code Language Models Remember? An Investigation on Data Extraction Attacks before and after Fine-tuning
  4. 2026Security in the Fine-Tuning Lifecycle of Large Language Models: Threats, Defenses,Evaluation, and Future Directions
  5. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  6. 2024Learning to Poison Large Language Models for Downstream Manipulation
  7. 2024Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directions
  8. 2024Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data