mlsanityData to 7 Oct 2026

CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation

arXiv 2407.07087 ↗First posted 9 Jul 2024cs.CL cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

Evaluating the degree of reproduction of copyright-protected content by language models (LMs) is of significant interest to the AI and legal communities. Although both literal and non-literal similarities are considered by courts when assessing the degree of reproduction, prior research has focused only on literal similarities. To bridge this gap, we introduce CopyBench, a benchmark designed to measure both literal and non-literal copying in LM generations. Using copyrighted fiction books as text sources, we provide automatic evaluation protocols to assess literal and non-literal copying, balanced against the model utility in terms of the ability to recall facts from the copyrighted works and generate fluent completions. We find that, although literal copying is relatively rare, two types of non-literal copying -- event copying and character copying -- occur even in models as small as 7B parameters. Larger models demonstrate significantly more copying, with literal copying rates increasing from 0.2% to 10.5% and non-literal copying from 2.3% to 5.9% when comparing Llama3-8B and 70B models, respectively. We further evaluate the effectiveness of current strategies for mitigating copying and show that (1) training-time alignment can reduce literal copying but may increase non-literal copying, and (2) current inference-time mitigation methods primarily reduce literal but not non-literal copying.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 31 of its 57 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

31 of 57 references.

References · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  8. 2023LIMA: Less Is More for Alignmentcited by 1,080
  9. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  10. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  11. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  12. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
19 more
  1. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 366
  2. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  3. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 198
  4. 2023Foundation Models and Fair Usecited by 112
  5. 2023On Provable Copyright Protection for Generative Modelscited by 97
  6. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 91
  7. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  8. 2023Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Taskscited by 53
  9. 2024Measuring and Controlling Instruction (In)Stability in Language Model Dialogscited by 51
  10. 2023Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuningcited by 48
  11. 2022Memorization in NLP Fine-tuning Methodscited by 43
  12. 2024Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMscited by 33
  13. 2024SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generationcited by 33
  14. 2022Submix: Practical Private Prediction for Large-Scale Language Modelscited by 31
  15. 2024Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenoncited by 30
  16. 2024AttributionBench: How Hard is Automatic Attribution Evaluation?cited by 26
  17. 2024CPR: Retrieval Augmented Generation for Copyright Protectioncited by 22
  18. 2022Privacy-Preserving Domain Adaptation of Semantic Parserscited by 12
  19. 2024Disguised Copyright Infringement of Latent Diffusion Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 16; 2026: 12.

35 papers.

Sort:

By area:

  1. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 88
  2. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 55
  3. 2025When One LLM Drools, Multi-LLM Collaboration Rulescited by 25
  4. 2024CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraintscited by 22
  5. 2024Exploring Memorization and Copyright Violation in Frontier LLMs: A Study of the New York Times v. OpenAI 2023 Lawsuitcited by 17
  6. 2025Position: Privacy Is Not Just Memorization!cited by 15
  7. 2024Measuring Copyright Risks of Large Language Model via Partial Information Probingcited by 8
  8. 2025The Surprising Effectiveness of Membership Inference with Simple N-Gram Coveragecited by 7
  9. 2025Extracting alignment data in open modelscited by 7
  10. 2024Copyright-Protected Language Generation via Adaptive Model Fusioncited by 6
  11. 2025ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Datacited by 6
  12. 2026Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Modelscited by 6
23 more
  1. 2025Position: Bridge the Gaps between Machine Unlearning and AI Regulationcited by 5
  2. 2025The KL3M Data Project: Copyright-Clean Training Resources for Large Language Modelscited by 5
  3. 2025Measuring LLM Novelty As The Frontier Of Original And High-Quality Outputcited by 5
  4. 2024Strong Copyright Protection for Language Models via Adaptive Model Fusioncited by 4
  5. 2025SUV: Scalable Large Language Model Copyright Compliance with Regularized Selective Unlearningcited by 4
  6. 2026Influential Training Data Retrieval for Explaining Verbalized Confidence of LLMscited by 4
  7. 2026Sparse Autoencoders are Capable LLM Jailbreak Mitigatorscited by 4
  8. 2025ISACL: Internal State Analyzer for Copyrighted Training Data Leakagecited by 2
  9. 2024The use of GPT-4o and Other Large Language Models for the Improvement and Design of Self-Assessment Scales for Measurement of Interpersonal Communication Skillscited by 1
  10. 2025Interrogating LLM design under a fair learning doctrinecited by 1
  11. 2025Beyond English: Unveiling Multilingual Bias in LLM Copyright Compliancecited by 1
  12. 2025Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Modelscited by 1
  13. 2026Computational Compliance for AI Regulation: Blueprint for a New Research Domaincited by 1
  14. 2026Copyright Detective: A Forensic System to Evidence LLMs Flickering Copyright Leakage Riskscited by 1
  15. 2026Erase Persona, Forget Lore: Benchmarking Multimodal Copyright Unlearning in Large Vision Language Modelscited by 1
  16. 2026Agentic Evaluation of Copyright Law Compliancecited by 1
  17. 2025Certified Mitigation of Worst-Case LLM Copyright Infringementcited by 0
  18. 2025Privacy Risks in Time Series Forecasting: User- and Record-Level Membership Inferencecited by 0
  19. 2026We Should Separate Memorization from Copyrightcited by 0
  20. 2026DP-RFT: Learning to Generate Synthetic Text via Differentially Private Reinforcement Fine-Tuningcited by 0
  21. 2026SiNFluD: Creating and Evaluating Figurative Language Dataset for Sindhicited by 0
  22. 2026Do Evaluation Metrics Detect Errors in Classical Chinese to English Translations?cited by 0
  23. 2026CopyShield: A Cross-Level Benchmark of Copyright Defenses in LLMscited by 0