mlsanityData to 7 Oct 2026

DE-COP: Detecting Copyrighted Content in Language Models Training Data

arXiv 2402.09910 ↗First posted 15 Feb 2024cs.CL cs.LGcited by 72

Not on the map. Why?

Cited by 30 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

How can we detect if copyrighted content was used in the training process of a language model, considering that the training data is typically undisclosed? We are motivated by the premise that a language model is likely to identify verbatim excerpts from its training text. We propose DE-COP, a method to determine whether a piece of copyrighted content was included in training. DE-COP's core approach is to probe an LLM with multiple-choice questions, whose options include both verbatim text and their paraphrases. We construct BookTection, a benchmark with excerpts from 165 books published prior and subsequent to a model's training cutoff, along with their paraphrases. Our experiments show that DE-COP surpasses the prior best method by 9.6% in detection performance (AUC) on models with logits available. Moreover, DE-COP also achieves an average accuracy of 72% for detecting suspect books on fully black-box models where prior methods give approximately 4% accuracy. The code and datasets are available at https://github.com/LeiLiLab/DE-COP.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 24 of its 32 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

24 of 32 references.

References · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2020Extracting Training Data from Large Language Modelscited by 2,144
  6. 2024Mixtral of Expertscited by 1,785
  7. 2021Membership Inference Attacks From First Principlescited by 839
  8. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  9. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  10. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  11. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  12. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
12 more
  1. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
  2. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
  3. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  4. 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 137
  5. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 123
  6. 2023Copyright Violations and Large Language Modelscited by 108
  7. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  8. 2023On Provable Copyright Protection for Generative Modelscited by 97
  9. 2018Understanding Membership Inferences on Well-Generalized Learning Modelscited by 83
  10. 2023Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Modelscited by 46
  11. 2023Can Copyright be Reduced to Privacy?cited by 24
  12. 2022Provably Confidential Language Modellingcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 31; 2026: 12.

72 papers.

Sort:

By area:

  1. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  2. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  3. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 102
  4. 2024Blind Baselines Beat Membership Inference Attacks for Foundation Modelscited by 65
  5. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoodscited by 61
  6. 2024Protecting Privacy in Multimodal Large Language Models with MLLMU-Benchcited by 61
  7. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 54
  8. 2024Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Methodcited by 54
  9. 2024Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Datacited by 49
  10. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  11. 2025A Survey on Data Contamination for Large Language Modelscited by 33
  12. 2024On the Challenges and Opportunities in Generative AIcited by 31
38 more
  1. 2024Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediationcited by 30
  2. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  3. 2025Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Modelscited by 25
  4. 2024Avoiding Copyright Infringement via Large Language Model Unlearningcited by 22
  5. 2024Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decodingcited by 20
  6. 2024Sampling-based Pseudo-Likelihood for Membership Inference Attackscited by 18
  7. 2024Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data?cited by 15
  8. 2024Detecting Training Data of Large Language Models via Expectation Maximizationcited by 13
  9. 2024LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakagecited by 13
  10. 2025How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergencecited by 13
  11. 2025DCR: Quantifying Data Contamination in LLMs Evaluationcited by 10
  12. 2024The Mosaic Memory of Large Language Modelscited by 9
  13. 2025Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Trainingcited by 9
  14. 2025SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attackscited by 9
  15. 2024Towards More Realistic Extraction Attacks: An Adversarial Perspectivecited by 8
  16. 2025Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Modelscited by 8
  17. 2024Fine-tuning can Help Detect Pretraining Data from Large Language Modelscited by 7
  18. 2024Creativity in AI: Progresses and Challengescited by 7
  19. 2025Unlocking Post-hoc Dataset Inference with Synthetic Datacited by 7
  20. 2025The Surprising Effectiveness of Membership Inference with Simple N-Gram Coveragecited by 7
  21. 2024Meta-Reasoning Improves Tool Use in Large Language Modelscited by 6
  22. 2025Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMscited by 6
  23. 2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
  24. 2025Leave No TRACE: Black-box Detection of Copyrighted Dataset Usage in Large Language Models via Watermarkingcited by 5
  25. 2024Using Large Language Models to Create AI Personas for Replication, Generalization and Prediction of Media Effects: An Empirical Test of 133 Published Experimental Research Findingscited by 4
  26. 2025On the Mutual Influence of Gender and Occupation in LLM Representationscited by 4
  27. 2025SUA: Stealthy Multimodal Large Language Model Unlearning Attackcited by 4
  28. 2025Beyond Public Access in LLM Pre-Training Datacited by 3
  29. 2025Membership Inference Attacks on Tokenizers of Large Language Modelscited by 3
  30. 2025Data Provenance Auditing of Fine-Tuned Large Language Models with a Text-Preserving Techniquecited by 3
  31. 2025Large Language Models as Search Engines: Societal Challengescited by 3
  32. 2024Quantifying Memorization and Detecting Training Data of Pre-trained Language Models using Japanese Newspapercited by 2
  33. 2024C2LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluationcited by 2
  34. 2024On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMscited by 2
  35. 2025Automatic Calibration for Membership Inference Attack on Large Language Modelscited by 2
  36. 2025LogitTrace: Detecting Benchmark Contamination via Layerwise Logit Trajectoriescited by 2
  37. 2026Membership Inference on LLMs in the Wildcited by 2
  38. 2024Many-Shot Regurgitation (MSR) Promptingcited by 1

50 most cited of 72 All 72 (JSON)