# CopyShield: A Cross-Level Benchmark of Copyright Defenses in LLMs

arXiv: https://arxiv.org/abs/2609.01161 · HTML: https://mlsanity.com/paper/2609.01161/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)

## Builds on (most cited first, 17 of 17)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [Who's Harry Potter? Approximate Unlearning in LLMs](https://mlsanity.com/paper/2310.02238/) (2023)
- [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://mlsanity.com/paper/2305.00118/) (2023)
- [Foundation Models and Fair Use](https://mlsanity.com/paper/2303.15715/) (2023)
- [Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy](https://mlsanity.com/paper/2210.17546/) (2022)
- [Evaluating Copyright Takedown Methods for Language Models](https://mlsanity.com/paper/2406.18664/) (2024)
- [CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation](https://mlsanity.com/paper/2407.07087/) (2024)
- [Extracting books from production language models](https://mlsanity.com/paper/2601.02671/) (2026)
- [Copyright-Protected Language Generation via Adaptive Model Fusion](https://mlsanity.com/paper/2412.06619/) (2024)
- [ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Data](https://mlsanity.com/paper/2504.14452/) (2025)
- [SCOPE: Intrinsic Semantic Space Control for Mitigating Copyright Infringement in LLMs](https://mlsanity.com/paper/2511.07001/) (2025)

## Abstract (arXiv)

Large language models can reproduce memorized text verbatim, yet copyright defenses are usually evaluated under incompatible protocols. We introduce CopyShield, a controlled benchmark comparing three representative defenses at distinct intervention levels: contrastive decoding (output), Direct Preference Optimization (behavioral), and activation intervention (representation). We evaluate CopyShield on two model families, LLaMA-3.1-8B and Mistral-7B-v0.3, using controlled memorization over five public-domain books and a shared protocol measuring literal leakage, calibrated non-literal leakage, utility, and degeneracy. Across these methods, intervention level is associated with distinct compliance-utility trade-offs. On LLaMA-3.1-8B, contrastive decoding remains near-degeneracy-free (0-2%) but reaches a literal-suppression floor at NV-Recall 0.192-0.203. DPO nearly eliminates literal leakage (0.263 to 0.002) but induces paraphrase-loop degeneracy in 58% of QA outputs, with no utility gain over the SFT baseline. Activation intervention attains the lowest non-literal flagging rate (1/200) by blocking 84% of non-literal queries before generation. Human evaluation confirms that DPO has low coherence, whereas activation lowers perceived copyright risk through broad refusal. On Mistral-7B-v0.3, the output- and representation-level patterns persist, while DPO degeneracy falls to 10-14%, showing that its severity is model-dependent. Together, CopyShield provides cross-level reference baselines and identifies targeted non-literal suppression as an open challenge. The code is available at https://github.com/spotai-mbzuai/CopyShield.git.