mlsanityData to 7 Oct 2026

The Privacy Fallacy of Crowdsourced Fine-Tuning: Extracting Proprietary Data via Topic-Based Poisoning

arXiv 2609.33985 ↗First posted 27 Sep 2026cs.CR cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Supervised fine-tuning (SFT) is widely used to adapt large language models to downstream tasks. Crowdsourcing user conversations is an established approach to collecting SFT data at scale while reducing the need for costly manual annotation. However, it also allows untrusted users to contribute data to the fine-tuning pipeline. We investigate an underexplored privacy risk arising from this setting: can a malicious user poison a small fraction of the crowdsourced data to amplify extraction of previously unseen instructions contributed by other users? We show that this is possible using only black-box, output-only access to the deployed model. Experiments across four models and two datasets demonstrate substantial increases in training-data extraction: with only 50 poisoned examples, near-verbatim extraction reaches 3.71× the rate without poisoning for Qwen2.5-14B on OpenMathInstruct and 3.08× for Llama-3.1-8B on AceReason. Data filtering also proves largely ineffective in detecting poisoned samples: even the best-performing method achieves only 0.378 in F-1 score, leaving the majority of poisoned samples undetected. These findings demonstrate that seemingly benign crowdsourced contributions can amplify leakage of other records while remaining difficult to identify through data filtering.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 45 of its 57 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

45 of 57 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2025Gemma 3 Technical Reportcited by 1,686
  8. 2021Membership Inference Attacks From First Principlescited by 839
  9. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  10. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  11. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  12. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
33 more
  1. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  2. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  3. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  4. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  5. 2023Poisoning Language Models During Instruction Tuningcited by 254
  6. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  7. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  8. 2024OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Datacited by 194
  9. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  10. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 100
  11. 2022Truth Serum: Poisoning Machine Learning Models to Reveal Their Secretscited by 94
  12. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learningcited by 85
  13. 2025Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samplescited by 66
  14. 2024Persistent Pre-Training Poisoning of LLMscited by 46
  15. 2022Membership Inference Attacks and Defenses in Neural Network Pruningcited by 42
  16. 2023The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Riskscited by 35
  17. 2022Amplifying Membership Exposure via Data Poisoningcited by 29
  18. 2024Teach LLMs to Phish: Stealing Private Information from Language Modelscited by 29
  19. 2021DATE: Detecting Anomalies in Text via Self-Supervision of Transformerscited by 25
  20. 2024Large Language Models for Anomaly and Out-of-Distribution Detection: A Surveycited by 24
  21. 2025Exploring the limits of strong membership inference attacks on large language modelscited by 20
  22. 2022Membership Inference Attacks and Generalization: A Causal Perspectivecited by 17
  23. 2025ShareChat: A Dataset of Chatbot Conversations in the Wildcited by 12
  24. 2023Early ChatGPT User Portrait through the Lens of Datacited by 6
  25. 2025Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoningcited by 6
  26. 2025Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMscited by 6
  27. 2025Be Careful When Fine-tuning On Open-Source LLMs: Your Fine-tuning Data Could Be Secretly Stolen!cited by 5
  28. 2025Text Anomaly Detection with Simplified Isolation Kernelcited by 5
  29. 2026Window-based Membership Inference Attacks Against Fine-tuned Large Language Modelscited by 5
  30. 2025Retracing the Past: LLMs Emit Training Data When They Get Lostcited by 3
  31. 2025PATCH: Mitigating PII Leakage in Language Models with Privacy-Aware Targeted Circuit PatcHingcited by 2
  32. 2026Discovering Universal Activation Directions for PII Leakage in Language Modelscited by 1
  33. 2026Loss Landscape Poisoning: Targeted Extraction of Unseen Training Data from LLMscited by 1

Built on by

Not cited yet.