mlsanityData to 7 Oct 2026

Do Language Models Plagiarize?

arXiv 2203.07618 ↗First posted 15 Mar 2022cs.CL cs.AIcited by 72

Not on the map. Why?

Abstract (arXiv)

Past literature has illustrated that language models (LMs) often memorize parts of training instances and reproduce them in natural language generation (NLG) processes. However, it is unclear to what extent LMs "reuse" a training corpus. For instance, models can generate paraphrased sentences that are contextually similar to training samples. In this work, therefore, we study three types of plagiarism (i.e., verbatim, paraphrase, and idea) among GPT-2 generated texts, in comparison to its training data, and further analyze the plagiarism patterns of fine-tuned LMs with domain-specific corpora which are extensively used in practice. Our results suggest that (1) three types of plagiarism widely exist in LMs beyond memorization, (2) both size and decoding methods of LMs are strongly associated with the degrees of plagiarism they exhibit, and (3) fine-tuned LMs' plagiarism patterns vary based on their corpus similarity and homogeneity. Given that a majority of LMs' training data is scraped from the Web without informing content owners, their reiteration of words, phrases, and even core ideas from training sets into generated texts has ethical implications. Their patterns are likely to exacerbate as both the size of LMs and their training data increase, raising concerns about indiscriminately pursuing larger models with larger training corpora. Plagiarized content can also contain individuals' personal and sensitive information. These findings overall cast doubt on the practicality of current LMs in mission-critical writing tasks and urge more discussions around the observed phenomena. Data and source code are available at https://github.com/Brit7777/LM-plagiarism.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 43 of its 97 references that are in the map. First posted on arXiv 15 Mar 2022.

Builds on

43 of 97 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2018Hierarchical Neural Story Generationcited by 1,182
  8. 2017A Closer Look at Memorization in Deep Networkscited by 936
  9. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  10. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  11. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  12. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
31 more
  1. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  2. 2019Automatic Detection of Generated Text is Easiest when Humans are Fooledcited by 260
  3. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  4. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  5. 2020CORD-19: The COVID-19 Open Research Datasetcited by 179
  6. 2019Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inferencecited by 168
  7. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  8. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  9. 2021TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generationcited by 118
  10. 2019Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learningcited by 117
  11. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 94
  12. 2020Toxicity Detection: Does Context Really Matter?cited by 93
  13. 2019Analyzing Information Leakage of Updates to Natural Language Modelscited by 89
  14. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86
  15. 2020Pre-training Multilingual Neural Machine Translation by Leveraging Alignment Informationcited by 64
  16. 2019Leveraging BERT for Extractive Text Summarization on Lecturescited by 63
  17. 2020A Non-Parametric Test to Detect Data-Copying in Generative Modelscited by 52
  18. 2019Patent Claim Generation by Fine-Tuning OpenAI GPT-2cited by 49
  19. 2021Artificial Text Detection via Examining the Topology of Attention Mapscited by 49
  20. 2020SongMASS: Automatic Song Writing with Pre-training and Alignment Constraintcited by 41
  21. 2018Towards Demystifying Membership Inference Attackscited by 39
  22. 2020Attacking Neural Text Detectorscited by 37
  23. 2021Privacy Regularization: Joint Privacy-Utility Optimization in Language Modelscited by 29
  24. 2021How BPE Affects Memorization in Transformerscited by 29
  25. 2021Detecting Fake News Using Machine Learning : A Systematic Literature Reviewcited by 19
  26. 2020Decoding Methods for Neural Narrative Generationcited by 18
  27. 2021Investigating Memorization of Conspiracy Theories in Text Generationcited by 15
  28. 2017A Deep Network Model for Paraphrase Detection in Short Text Messagescited by 9
  29. 2019An Iterative Polishing Framework based on Quality Aware Masked Language Model for Chinese Poetry Generationcited by 9
  30. 2020Multilevel Text Alignment with Cross-Document Attentioncited by 7
  31. 2014Plagiarism Detection on Electronic Text based Assignments using Vector Space Model (ICIAfS14)cited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 17; 2024: 26; 2025: 14; 2026: 11.

72 papers.

Sort:

By area:

  1. 2023Large Language Model Unlearningcited by 288
  2. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  3. 2023Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvaturecited by 278
  4. 2023Understanding and Mitigating Copying in Diffusion Modelscited by 205
  5. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  6. 2023Foundation Models and Fair Usecited by 112
  7. 2023A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directionscited by 101
  8. 2023On Provable Copyright Protection for Generative Modelscited by 97
  9. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  10. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  11. 2024A Design Space for Intelligent and Interactive Writing Assistantscited by 76
  12. 2024On the Societal Impact of Open Foundation Modelscited by 64
38 more
  1. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  2. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 54
  3. 2024Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMscited by 50
  4. 2023Exploring Memorization in Fine-tuned Language Modelscited by 44
  5. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  6. 2023GPT-who: An Information Density-based Machine-Generated Text Detectorcited by 38
  7. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 36
  8. 2024Copyright Protection in Generative AI: A Technical Perspectivecited by 35
  9. 2025Towards Label-Only Membership Inference Attack against Pre-trained Large Language Modelscited by 34
  10. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  11. 2024Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detectioncited by 22
  12. 2023Does Human Collaboration Enhance the Accuracy of Identifying LLM-Generated Deepfake Texts?cited by 21
  13. 2023An Evaluation on Large Language Model Outputs: Discourse and Memorizationcited by 20
  14. 2023Citation: A Key to Building Responsible and Accountable Large Language Modelscited by 20
  15. 2024Causal Estimation of Memorisation Profilescited by 20
  16. 2025AdaDetectGPT: Adaptive Detection of LLM-Generated Text with Statistical Guaranteescited by 20
  17. 2022Reward Reports for Reinforcement Learningcited by 18
  18. 2022Noise-Robust De-Duplication at Scalecited by 17
  19. 2025Analyzing Memorization in Large Language Models through the Lens of Model Attributioncited by 14
  20. 2024Privacy-Aware Visual Language Modelscited by 12
  21. 2024DALD: Improving Logits-based Detector without Logits from Black-box LLMscited by 12
  22. 2024Online Detection of LLM-Generated Texts via Sequential Hypothesis Testing by Bettingcited by 11
  23. 2023TOPFORMER: Topology-Aware Authorship Attribution of Deepfake Texts with Diverse Writing Stylescited by 10
  24. 2024ReMoDetect: Reward Models Recognize Aligned LLM's Generationscited by 10
  25. 2024MCP-Solver: Integrating Language Models with Constraint Programming Systemscited by 9
  26. 2025Human Texts Are Outliers: Detecting LLM-generated Texts via Out-of-distribution Detectioncited by 8
  27. 2024Topic-Based Watermarks for Large Language Modelscited by 7
  28. 2024U Can't Gen This? A Survey of Intellectual Property Protection Methods for Data in Generative AIcited by 7
  29. 2025Extracting alignment data in open modelscited by 7
  30. 2022The Infinite Index: Information Retrieval on Generative Text-To-Image Modelscited by 6
  31. 2024Will GPT-4 Run DOOM?cited by 6
  32. 2024The Unreasonable Ineffectiveness of Nucleus Sampling on Mitigating Text Memorizationcited by 6
  33. 2025ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Datacited by 6
  34. 2024Newswire: A Large-Scale Structured Database of a Century of Historical Newscited by 5
  35. 2024Awes, Laws, and Flaws From Today's LLM Researchcited by 5
  36. 2025The Thin Line Between Comprehension and Persuasion in LLMscited by 5
  37. 2023"I'd Like to Have an Argument, Please": Argumentative Reasoning in Large Language Modelscited by 4
  38. 2026AttenMIA: LLM Membership Inference Attack through Attention Signalscited by 4

50 most cited of 72 All 72 (JSON)