mlsanityData to 7 Oct 2026

Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!

arXiv 2505.15656 ↗First posted 21 May 2025cs.CLcited by 5

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Fine-tuning on open-source Large Language Models (LLMs) with proprietary data is now a standard practice for downstream developers to obtain task-specific LLMs. Surprisingly, we reveal a new and concerning risk along with the practice: the creator of the open-source LLMs can later extract the private downstream fine-tuning data through simple backdoor training, only requiring black-box access to the fine-tuned downstream model. Our comprehensive experiments, across 4 popularly used open-source models with 3B to 32B parameters and 2 downstream datasets, suggest that the extraction performance can be strikingly high: in practical settings, as much as 76.3% downstream fine-tuning data (queries) out of a total 5,000 samples can be perfectly extracted, and the success rate can increase to 94.9% in more ideal settings. We also explore a detection-based defense strategy but find it can be bypassed with improved attack. Overall, we highlight the emergency of this newly identified data breaching risk in fine-tuning, and we hope that more follow-up research could push the progress of addressing this concerning risk. The code and data used in our experiments are released at https://github.com/thu-coai/Backdoor-Data-Extraction.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 46 of its 56 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

46 of 56 references.

References · 46

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2016Deep Learning with Differential Privacycited by 2,682
  7. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2020Knowledge Distillation: A Surveycited by 1,578
  10. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  12. 2022Quantifying Memorization Across Neural Language Modelscited by 755
34 more
  1. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  2. 2019STRIP: A Defence Against Trojan Attacks on Deep Neural Networkscited by 437
  3. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  4. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  5. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  6. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  7. 2020Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defensescited by 171
  8. 2021Adversarial Unlearning of Backdoors via Implicit Hypergradientcited by 160
  9. 2017Neural Trojanscited by 151
  10. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 143
  11. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 100
  12. 2022Truth Serum: Poisoning Machine Learning Models to Reveal Their Secretscited by 94
  13. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 91
  14. 2022A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarkscited by 70
  15. 2022Backdoor Defense with Machine Unlearningcited by 61
  16. 2022BadPrompt: Backdoor Attacks on Continuous Promptscited by 59
  17. 2022Fine-mixing: Mitigating Backdoors in Fine-tuned Language Modelscited by 50
  18. 2022BITE: Textual Backdoor Attacks with Iterative Trigger Injectioncited by 47
  19. 2024Revisiting Backdoor Attacks against Large Vision-Language Models from Domain Shiftcited by 45
  20. 2023Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimationcited by 36
  21. 2025Towards Label-Only Membership Inference Attack against Pre-trained Large Language Modelscited by 34
  22. 2024Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Modelscited by 28
  23. 2024Is poisoning a real threat to LLM alignment? Maybe more so than you thinkcited by 27
  24. 2025Extracting memorized pieces of (copyrighted) books from open-weight language modelscited by 27
  25. 2023From Shortcuts to Triggers: Backdoor Defense with Denoised PoEcited by 26
  26. 2024Here's a Free Lunch: Sanitizing Backdoored Models with Model Mergecited by 21
  27. 2023Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensive Demonstrationscited by 20
  28. 2024PreCurious: How Innocent Pre-Trained Language Models Turn into Privacy Trapscited by 20
  29. 2024Privacy Backdoors: Stealing Data with Corrupted Pretrained Modelscited by 20
  30. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 14
  31. 2025Tokens for Learning, Tokens for Unlearning: Mitigating Membership Inference Attacks in Large Language Models via Dual-Purpose Trainingcited by 14
  32. 2025AISafetyLab: A Comprehensive Framework for AI Safety Evaluation and Improvementcited by 11
  33. 2024Two Heads are Better than One: Nested PoE for Robust Defense Against Multi-Backdoorscited by 6
  34. 2025Can Differentially Private Fine-tuning LLMs Protect Against Privacy Attacks?cited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuningcited by 4
  2. 2025Offline and Online KL-Regularized RLHF under Differential Privacycited by 3
  3. 2026The Missing Half: Unveiling Training-time Implicit Safety Risks Beyond Deploymentcited by 0
  4. 2026Secret Stealing Attacks on Local LLM Fine-Tuning through Supply-Chain Model Code Backdoorscited by 0
  5. 2026The Privacy Fallacy of Crowdsourced Fine-Tuning: Extracting Proprietary Data via Topic-Based Poisoningcited by 0