mlsanityData to 14 Oct 2026

Exploring Memorization and Copyright Violation in Frontier LLMs: A Study of the New York Times v. OpenAI 2023 Lawsuit

arXiv 2412.06370 ↗First posted 9 Dec 2024cs.LG cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

Copyright infringement in frontier LLMs has received much attention recently due to the New York Times v. OpenAI lawsuit, filed in December 2023. The New York Times claims that GPT-4 has infringed its copyrights by reproducing articles for use in LLM training and by memorizing the inputs, thereby publicly displaying them in LLM outputs. Our work aims to measure the propensity of OpenAI's LLMs to exhibit verbatim memorization in its outputs relative to other LLMs, specifically focusing on news articles. We discover that both GPT and Claude models use refusal training and output filters to prevent verbatim output of the memorized articles. We apply a basic prompt template to bypass the refusal training and show that OpenAI models are currently less prone to memorization elicitation than models from Meta, Mistral, and Anthropic. We find that as models increase in size, especially beyond 100 billion parameters, they demonstrate significantly greater capacity for memorization. Our findings have practical implications for training: more attention must be placed on preventing verbatim memorization in very large models. Our findings also have legal significance: in assessing the relative memorization capacity of OpenAI's LLMs, we probe the strength of The New York Times's copyright infringement claims and OpenAI's legal defenses, while underscoring issues at the intersection of generative AI, law, and policy.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 16 of its 39 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

16 of 39 references.

References · 16

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  4. 2020Extracting Training Data from Large Language Modelscited by 2,189
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  6. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  7. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  8. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 618
  9. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  10. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  11. 2023Large Language Model Unlearningcited by 302
  12. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 163
4 more
  1. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 123
  2. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 43
  3. 2024CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generationcited by 35
  4. 2024Many-Shot Regurgitation (MSR) Promptingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 4.

18 papers.

Sort:

By area:

  1. 2025Language Models May Verbatim Complete Text They Were Not Explicitly Trained Oncited by 21
  2. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 10
  3. 2025Extracting alignment data in open modelscited by 8
  4. 2026G-Drift MIA: Membership Inference via Gradient-Induced Feature Drift in LLMscited by 8
  5. 2025The Surprising Effectiveness of Membership Inference with Simple N-Gram Coveragecited by 7
  6. 2025From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMscited by 6
  7. 2024ExpShield: Safeguarding Web Text from Unauthorized Crawling and LLM Exploitationcited by 3
  8. 2025Align-then-Unlearn: Embedding Alignment for LLM Unlearningcited by 3
  9. 2025A Lightweight Method to Disrupt Memorized Sequences in LLMcited by 2
  10. 2025Free and Fair Hardware: A Pathway to Copyright Infringement-Free Verilog Generation using LLMscited by 2
  11. 2025Positional Fragility in LLMs: How Offset Effects Reshape Our Understanding of Memorization Riskscited by 2
  12. 2026Erase Persona, Forget Lore: Benchmarking Multimodal Copyright Unlearning in Large Vision Language Modelscited by 2
6 more
  1. 2025Interrogating LLM design under a fair learning doctrinecited by 1
  2. 2025Rethinking Memorization Measures and their Implications in Large Language Modelscited by 1
  3. 2025Forgetting-MarI: LLM Unlearning via Marginal Information Regularizationcited by 1
  4. 2025Copyright Detection in Large Language Models: An Ethical Approach to Generative AI Developmentcited by 0
  5. 2026Responsible AI for General-Purpose Systems: Overview, Challenges, and A Path Forwardcited by 0
  6. 2026We Should Separate Memorization from Copyrightcited by 0