mlsanityData to 14 Oct 2026

Evaluating Copyright Takedown Methods for Language Models

arXiv 2406.18664 ↗First posted 26 Jun 2024cs.CL cs.LGcited by 43

Not on the map. Why?

Cited by 22 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language models (LMs) derive their capabilities from extensive training on diverse data, including potentially copyrighted material. These models can memorize and generate content similar to their training data, posing potential concerns. Therefore, model creators are motivated to develop mitigation methods that prevent generating protected content. We term this procedure as copyright takedowns for LMs, noting the conceptual similarity to (but legal distinction from) the DMCA takedown This paper introduces the first evaluation of the feasibility and side effects of copyright takedowns for LMs. We propose CoTaEval, an evaluation framework to assess the effectiveness of copyright takedown methods, the impact on the model's ability to retain uncopyrightable factual knowledge from the training data whose recitation is embargoed, and how well the model maintains its general utility and efficiency. We examine several strategies, including adding system prompts, decoding-time filtering interventions, and unlearning approaches. Our findings indicate that no tested method excels across all metrics, showing significant room for research in this unique problem setting and indicating potential unresolved challenges for live policy proposals.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 51 of its 83 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

51 of 83 references.

References · 51

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  8. 2020Extracting Training Data from Large Language Modelscited by 2,189
  9. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  11. 2022Editing Models with Task Arithmeticcited by 1,120
  12. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
38 more
  1. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  2. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  3. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 618
  4. 2019Certified Data Removal from Machine Learning Modelscited by 535
  5. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 533
  6. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  7. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 504
  8. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  9. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  10. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  11. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 381
  12. 2016NewsQA: A Machine Comprehension Datasetcited by 331
  13. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  14. 2021Unrolling SGD: Understanding Factors Influencing Machine Unlearningcited by 279
  15. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  16. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 224
  17. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 195
  18. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  19. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 158
  20. 2021Counterfactual Memorization in Neural Language Modelscited by 156
  21. 2022Continual Learning and Private Unlearningcited by 130
  22. 2023What's In My Big Data?cited by 129
  23. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  24. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 125
  25. 2023Foundation Models and Fair Usecited by 116
  26. 2023Copyright Violations and Large Language Modelscited by 108
  27. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 100
  28. 2023DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Modelscited by 79
  29. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  30. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  31. 2023The Foundation Model Transparency Indexcited by 66
  32. 2024Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMscited by 50
  33. 2023Talkin' 'Bout AI Generation: Copyright and the Generative-AI Supply Chaincited by 46
  34. 2024Proving membership in LLM pretraining data via data watermarkscited by 40
  35. 2024SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generationcited by 34
  36. 2023Data Portraits: Recording Foundation Model Training Datacited by 30
  37. 2023How to Protect Copyright Data in Optimization of Large Language Models?cited by 26
  38. 2024Fantastic Copyrighted Beasts and How (Not) to Generate Themcited by 24

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 23; 2026: 8.

43 papers.

Sort:

By area:

  1. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 261
  2. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 123
  3. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  4. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 89
  5. 2024Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Datacited by 50
  6. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 28
  7. 2024Towards Robust Knowledge Unlearning: An Adversarial Framework for Assessing and Improving Unlearning Robustness in Large Language Modelscited by 27
  8. 2025NeuralOS: Towards Simulating Operating Systems via Neural Generative Modelscited by 19
  9. 2024Exploring Memorization and Copyright Violation in Frontier LLMs: A Study of the New York Times v. OpenAI 2023 Lawsuitcited by 18
  10. 2024Unlearn and Burn: Adversarial Machine Unlearning Requests Destroy Model Accuracycited by 17
  11. 2024Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Researchcited by 15
  12. 2025Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awarenesscited by 14
31 more
  1. 2025Distillation Robustifies Unlearningcited by 11
  2. 2025Improving LLM Unlearning Robustness via Random Perturbationscited by 9
  3. 2025R-TOFU: Unlearning in Large Reasoning Modelscited by 9
  4. 2024Copyright-Protected Language Generation via Adaptive Model Fusioncited by 7
  5. 2025ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Datacited by 6
  6. 2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
  7. 2025Obliviate: Efficient Unmemorization for Protecting Intellectual Property in Large Language Modelscited by 5
  8. 2025SUV: Scalable Large Language Model Copyright Compliance with Regularized Selective Unlearningcited by 5
  9. 2024Strong Copyright Protection for Language Models via Adaptive Model Fusioncited by 4
  10. 2024Negative Token Merging: Image-based Adversarial Feature Guidancecited by 4
  11. 2025DUSK: Do Not Unlearn Shared Knowledgecited by 4
  12. 2025A Survey on Unlearning in Large Language Modelscited by 4
  13. 2025Towards Benchmarking Privacy Vulnerabilities in Selective Forgetting with Large Language Modelscited by 4
  14. 2025Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detectioncited by 3
  15. 2025Tasks and Roles in Legal AI: Data Curation, Annotation, and Verificationcited by 3
  16. 2025SCOPE: Intrinsic Semantic Space Control for Mitigating Copyright Infringement in LLMscited by 3
  17. 2025Beyond English: Unveiling Multilingual Bias in LLM Copyright Compliancecited by 2
  18. 2025Machine Text Detectors are Membership Inference Attackscited by 2
  19. 2024Inner-Probe: Discovering Copyright-related Data Generation in LLM Architecturecited by 1
  20. 2025Interrogating LLM design under a fair learning doctrinecited by 1
  21. 2025Certified Mitigation of Worst-Case LLM Copyright Infringementcited by 1
  22. 2025Towards Mitigating Excessive Forgetting in LLM Unlearning via Entanglement-Guidance with Proxy Constraintcited by 1
  23. 2026Rethinking Benign Relearning: Syntax as the Hidden Driver of Unlearning Failurescited by 1
  24. 2026Unsafe2Safe: Controllable Image Anonymization for Downstream Utilitycited by 1
  25. 2026Agentic Evaluation of Copyright Law Compliancecited by 1
  26. 2026GROM: Gradient-Free Rapid One-Shot Machine Unlearningcited by 1
  27. 2025Nine Ways to Break Copyright Law and Why Our LLM Won't: A Fair Use Aligned Generation Frameworkcited by 0
  28. 2026EvoMU: Evolutionary Machine Unlearningcited by 0
  29. 2026We Should Separate Memorization from Copyrightcited by 0
  30. 2026On the Recoverability of Private Information Unlearning in Large Language Modelscited by 0
  31. 2026CopyShield: A Cross-Level Benchmark of Copyright Defenses in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Avoiding Copyright Infringement via Large Language Model Unlearning
  2. 2024Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data?
  3. 2024Purifying Large Language Models by Ensembling a Small Language Model
  4. 2024Investigating the Feasibility of Mitigating Potential Copyright Infringement via Large Language Model Unlearning
  5. 2025Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends
  6. 2025Bridging the Copyright Gap: Do Large Vision-Language Models Recognize and Respect Copyrighted Content?
  7. 2026Extracting books from production language models
  8. 2025A Statistical Hypothesis Testing Framework for Data Misappropriation Detection in Large Language Models