mlsanityData to 7 Oct 2026

How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN

arXiv 2111.09509 ↗First posted 18 Nov 2021cs.CLcited by 124

Not on the map. Why?

Abstract (arXiv)

Current language models can generate high-quality text. Are they simply copying text they have seen before, or have they learned generalizable linguistic abstractions? To tease apart these possibilities, we introduce RAVEN, a suite of analyses for assessing the novelty of generated text, focusing on sequential structure (n-grams) and syntactic structure. We apply these analyses to four neural language models (an LSTM, a Transformer, Transformer-XL, and GPT-2). For local structure - e.g., individual dependencies - model-generated text is substantially less novel than our baseline of human-generated text from each model's test set. For larger-scale structure - e.g., overall sentence structure - model-generated text is as novel or even more novel than the human-generated baseline, but models still sometimes copy substantially, in some cases duplicating passages over 1,000 words long from the training set. We also perform extensive manual analysis showing that GPT-2's novel text is usually well-formed morphologically and syntactically but has reasonably frequent semantic issues (e.g., being self-contradictory).

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Builds on 49 of its 102 references that are in the map. First posted on arXiv 18 Nov 2021.

Builds on

49 of 102 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  11. 2020Extracting Training Data from Large Language Modelscited by 2,144
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
37 more
  1. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  2. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  3. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  4. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  5. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  6. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  7. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  8. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  9. 2020Evaluation of Text Generation: A Surveycited by 275
  10. 2016Using the Output Embedding to Improve Language Modelscited by 266
  11. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  12. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  13. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  14. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  15. 2017Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluationcited by 206
  16. 2018Constituency Parsing with a Self-Attentive Encodercited by 196
  17. 2020The Next Decade in AI: Four Steps Towards Robust Artificial Intelligencecited by 194
  18. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  19. 2018Dialogue Natural Language Inferencecited by 135
  20. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  21. 2019Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Trainingcited by 110
  22. 2016Recurrent Neural Network Grammarscited by 109
  23. 2020PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Trackingcited by 104
  24. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  25. 2019Do Massively Pretrained Language Models Make Better Storytellers?cited by 90
  26. 2021Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Textcited by 89
  27. 2016Improving Neural Language Models with a Continuous Cachecited by 81
  28. 2021Language Model Evaluation Beyond Perplexitycited by 77
  29. 2020Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networkscited by 72
  30. 2016Efficient softmax approximation for GPUscited by 67
  31. 2021Frequency Effects on Syntactic Rule Learning in Transformerscited by 59
  32. 2020Efficient Second-Order TreeCRF for Neural Dependency Parsingcited by 48
  33. 2019Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languagescited by 37
  34. 2019Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solvingcited by 34
  35. 2021On Compositional Generalization of Neural Machine Translationcited by 34
  36. 2021Distinguishing rule- and exemplar-based generalization in learning systemscited by 12
  37. 2020Recurrent babbling: evaluating the acquisition of grammar from limited input datacited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 26; 2024: 38; 2025: 37; 2026: 10.

124 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  3. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  4. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  5. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  6. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  7. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  8. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  9. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 97
  10. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  11. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 80
  12. 2024Delving into LLM-assisted writing in biomedical publications through excess vocabularycited by 75
38 more
  1. 2023AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZcited by 74
  2. 2022Do Language Models Plagiarize?cited by 72
  3. 2024Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNscited by 65
  4. 2023The Next Chapter: A Study of Large Language Models in Storytellingcited by 58
  5. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  6. 2023On the Creativity of Large Language Modelscited by 57
  7. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  8. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 50
  9. 2024UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Modelscited by 49
  10. 2024Echoes in AI: Quantifying lack of plot diversity in LLM outputscited by 47
  11. 2024Detection and Measurement of Syntactic Templates in Generated Textcited by 44
  12. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  13. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  14. 2022ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Datacited by 34
  15. 2024Generative AI for Synthetic Data Generation: Methods, Challenges and the Futurecited by 31
  16. 2024On the Privacy Risk of In-context Learningcited by 31
  17. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  18. 2024Localizing Paragraph Memorization in Language Modelscited by 30
  19. 2022Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russiancited by 29
  20. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  21. 2023Dissociating language and thought in large language modelscited by 27
  22. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  23. 2025Protecting Users From Themselves: Safeguarding Contextual Privacy in Interactions with Conversational Agentscited by 26
  24. 2024Evaluating n-Gram Novelty of Language Models Using Rusty-DAWGcited by 24
  25. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  26. 2025Sparks of Science: Hypothesis Generation Using Structured Paper Datacited by 22
  27. 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19
  28. 2023Impact of Co-occurrence on Factual Knowledge of Large Language Modelscited by 19
  29. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 19
  30. 2022Evaluating Distributional Distortion in Neural Language Modelingcited by 18
  31. 2022You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LMcited by 18
  32. 2023Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Modelscited by 18
  33. 2023The Ethical Implications of Generative Audio Models: A Systematic Literature Reviewcited by 18
  34. 2024Sampling-based Pseudo-Likelihood for Membership Inference Attackscited by 18
  35. 2024Evaluating Diversity in Automatic Poetry Generationcited by 17
  36. 2023Exploring the Robustness of Large Language Models for Solving Programming Problemscited by 16
  37. 2024Reconstruct Your Previous Conversations! Comprehensively Investigating Privacy Leakage Risks in Conversations with GPT Modelscited by 16
  38. 2024Evaluating Morphological Compositional Generalization in Large Language Modelscited by 16

50 most cited of 124 All 124 (JSON)