mlsanityData to 7 Oct 2026

Strong Copyright Protection for Language Models via Adaptive Model Fusion

arXiv 2407.20105 ↗First posted 29 Jul 2024cs.LG cs.CRcited by 4

Not on the map. Why?

Abstract (arXiv)

The risk of language models unintentionally reproducing copyrighted material from their training data has led to the development of various protective measures. In this paper, we propose model fusion as an effective solution to safeguard against copyright infringement. In particular, we introduce Copyright-Protecting Fusion (CP-Fuse), an algorithm that adaptively combines language models to minimize the reproduction of protected materials. CP-Fuse is inspired by the recently proposed Near-Access Free (NAF) framework and additionally incorporates a desirable balancing property that we demonstrate prevents the reproduction of memorized training data. Our results show that CP-Fuse significantly reduces the memorization of copyrighted content while maintaining high-quality text and code generation. Furthermore, we demonstrate how CP-Fuse can be integrated with other techniques for enhanced protection.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 52 of its 65 references that are in the map. First posted on arXiv 29 Jul 2024.

Builds on

52 of 65 references.

References · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2016Deep Learning with Differential Privacycited by 2,682
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  9. 2019Machine Unlearningcited by 1,020
  10. 2021Measuring Coding Challenge Competence With APPScited by 879
  11. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  12. 2023StarCoder: may the source be with you!cited by 790
38 more
  1. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  3. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  4. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  5. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  6. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  7. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  8. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 366
  9. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  10. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  11. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  12. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  13. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  14. 2020Mixed-Privacy Forgetting in Deep Networkscited by 158
  15. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  16. 2021Large-Scale Differentially Private BERTcited by 117
  17. 2023Foundation Models and Fair Usecited by 112
  18. 2023Copyright Violations and Large Language Modelscited by 108
  19. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
  20. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 97
  21. 2023On Provable Copyright Protection for Generative Modelscited by 97
  22. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  23. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  24. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  25. 2023Did the Neurons Read your Book? Document-level Membership Inference for Large Language Modelscited by 51
  26. 2024Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMscited by 50
  27. 2023Talkin' 'Bout AI Generation: Copyright and the Generative-AI Supply Chaincited by 45
  28. 2023Fusing Models with Complementary Expertisecited by 45
  29. 2022Memorization in NLP Fine-tuning Methodscited by 43
  30. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 42
  31. 2022Privacy Adhering Machine Un-learning in NLPcited by 35
  32. 2024CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generationcited by 35
  33. 2024SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generationcited by 33
  34. 2023Privacy Implications of Retrieval-Based Language Modelscited by 31
  35. 2023SAFE: Machine Unlearning With Shard Graphscited by 28
  36. 2023How to Protect Copyright Data in Optimization of Large Language Models?cited by 26
  37. 2024Purifying Large Language Models by Ensembling a Small Language Modelcited by 25
  38. 2023Can Copyright be Reduced to Privacy?cited by 24

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2025Merge to Mix: Mixing Datasets via Model Mergingcited by 7
  3. 2024Copyright-Protected Language Generation via Adaptive Model Fusioncited by 6
  4. 2025SUV: Scalable Large Language Model Copyright Compliance with Regularized Selective Unlearningcited by 4